[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 551712144 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003284] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008330] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011138] LSM: Security Framework initializing [ 0.012057] Yama: becoming mindful. [ 0.013035] SELinux: Initializing. [ 0.014073] *** VALIDATE selinux *** [ 0.023038] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028404] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032039] *** VALIDATE tmpfs *** [ 0.033455] *** VALIDATE proc *** [ 0.035158] *** VALIDATE cgroup *** [ 0.036009] *** VALIDATE cgroup2 *** [ 0.037297] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040041] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.043665] debug: unmapping init [mem 0xffffffffa8a59000-0xffffffffa8a60fff] [ 0.045166] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046727] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047029] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051028] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054321] rcu: Hierarchical SRCU implementation. [ 0.056429] smp: Bringing up secondary CPUs ... [ 0.057615] x86: Booting SMP configuration: [ 0.058028] .... node #0, CPUs: #1 #2 #3 [ 0.062097] smp: Brought up 1 node, 4 CPUs [ 0.064019] smpboot: Max logical packages: 1 [ 0.065015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142526] node 0 deferred pages initialised in 75ms [ 0.147096] devtmpfs: initialized [ 0.148267] x86/mm: Memory block size: 128MB [ 0.152163] gcov: version magic: 0x41383552 [ 0.155302] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156129] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158447] pinctrl core: initialized pinctrl subsystem [ 0.161253] [ 0.161915] ************************************************************* [ 0.164015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167015] ** ** [ 0.169012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172015] ** ** [ 0.174012] ** This means that this kernel is built to expose internal ** [ 0.177014] ** IOMMU data structures, which may compromise security on ** [ 0.179013] ** your system. ** [ 0.182013] ** ** [ 0.184013] ** If you see this message and you are not debugging the ** [ 0.187021] ** kernel, report this immediately to your vendor! ** [ 0.189014] ** ** [ 0.192012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194014] ************************************************************* [ 0.197607] NET: Registered protocol family 16 [ 0.199505] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.205066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.209108] cpuidle: using governor menu [ 0.210621] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.213427] PCI: Using configuration type 1 for base access [ 0.215179] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.224148] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.225020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.226588] cryptd: max_cpu_qlen set to 1000 [ 0.228300] ACPI: Added _OSI(Module Device) [ 0.229000] ACPI: Added _OSI(Processor Device) [ 0.229000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236614] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243349] ACPI: Interpreter enabled [ 0.245075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.247012] ACPI: Using IOAPIC for interrupt routing [ 0.249101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253419] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266754] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.275275] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.280688] acpiphp: Slot [2] registered [ 0.282094] acpiphp: Slot [5] registered [ 0.284171] acpiphp: Slot [6] registered [ 0.285135] acpiphp: Slot [7] registered [ 0.286156] acpiphp: Slot [8] registered [ 0.288123] acpiphp: Slot [9] registered [ 0.290202] acpiphp: Slot [10] registered [ 0.291121] acpiphp: Slot [3] registered [ 0.293110] acpiphp: Slot [4] registered [ 0.294109] acpiphp: Slot [11] registered [ 0.296104] acpiphp: Slot [12] registered [ 0.297165] acpiphp: Slot [13] registered [ 0.299113] acpiphp: Slot [14] registered [ 0.301114] acpiphp: Slot [15] registered [ 0.302094] acpiphp: Slot [16] registered [ 0.303078] acpiphp: Slot [17] registered [ 0.304130] acpiphp: Slot [18] registered [ 0.305103] acpiphp: Slot [19] registered [ 0.307111] acpiphp: Slot [20] registered [ 0.308196] acpiphp: Slot [21] registered [ 0.310151] acpiphp: Slot [22] registered [ 0.312113] acpiphp: Slot [23] registered [ 0.314132] acpiphp: Slot [24] registered [ 0.315166] acpiphp: Slot [25] registered [ 0.317109] acpiphp: Slot [26] registered [ 0.319171] acpiphp: Slot [27] registered [ 0.321153] acpiphp: Slot [28] registered [ 0.322165] acpiphp: Slot [29] registered [ 0.324122] acpiphp: Slot [30] registered [ 0.326111] acpiphp: Slot [31] registered [ 0.327061] PCI host bridge to bus 0000:00 [ 0.329019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.334026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.337027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.340048] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.342033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348086] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352367] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.368015] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.380140] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.382924] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.385045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387935] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.409033] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.414017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.420587] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.428017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.434018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.451019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.461397] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.469018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.477019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.499014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.507032] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.516016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.522012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.543016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.555515] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.567018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.574021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.590019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.604908] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.612015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.618015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.641018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.651117] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.658018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.665018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.677017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.688580] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.691354] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.694417] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.697283] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.700268] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.705123] iommu: Default domain type: Passthrough [ 0.706260] SCSI subsystem initialized [ 0.708139] ACPI: bus type USB registered [ 0.709116] usbcore: registered new interface driver usbfs [ 0.711103] usbcore: registered new interface driver hub [ 0.713085] usbcore: registered new device driver usb [ 0.715174] pps_core: LinuxPPS API ver. 1 registered [ 0.717012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.720052] PTP clock support registered [ 0.721066] EDAC MC: Ver: 3.0.0 [ 0.723143] PCI: Using ACPI for IRQ routing [ 0.724364] NetLabel: Initializing [ 0.726011] NetLabel: domain hash size = 128 [ 0.727010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.729080] NetLabel: unlabeled traffic allowed by default [ 0.732299] vgaarb: loaded [ 0.733201] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.735011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.741000] clocksource: Switched to clocksource kvm-clock [ 0.839862] VFS: Disk quotas dquot_6.6.0 [ 0.841731] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.843924] *** VALIDATE ramfs *** [ 0.845092] *** VALIDATE hugetlbfs *** [ 0.846071] pnp: PnP ACPI init [ 0.848348] pnp: PnP ACPI: found 6 devices [ 0.876555] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.879245] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.880731] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.882726] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.885372] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.887894] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.891166] NET: Registered protocol family 2 [ 0.893822] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.899701] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.903614] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.909466] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.912913] TCP: Hash tables configured (established 65536 bind 65536) [ 0.916115] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.919517] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.922558] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.925678] NET: Registered protocol family 1 [ 0.928368] RPC: Registered named UNIX socket transport module. [ 0.929804] RPC: Registered udp transport module. [ 0.930899] RPC: Registered tcp transport module. [ 0.932527] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.934749] NET: Registered protocol family 44 [ 0.936152] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.938199] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.940399] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.942674] PCI: CLS 0 bytes, default 64 [ 0.945555] Unpacking initramfs... [ 2.335653] debug: unmapping init [mem 0xffff9b637cc54000-0xffff9b637ffbffff] [ 2.339747] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.342231] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.345154] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.893820] Initialise system trusted keyrings [ 2.895630] Key type blacklist registered [ 2.897688] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.910516] zbud: loaded [ 2.913874] *** VALIDATE nfs *** [ 2.916327] *** VALIDATE nfs4 *** [ 2.917972] pstore: using deflate compression [ 2.921514] Platform Keyring initialized [ 3.018993] NET: Registered protocol family 38 [ 3.020403] Key type asymmetric registered [ 3.022727] Asymmetric key parser 'x509' registered [ 3.024911] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.028265] io scheduler mq-deadline registered [ 3.030339] io scheduler kyber registered [ 3.032131] io scheduler bfq registered [ 3.035125] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.038412] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.041483] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.044328] ACPI: Power Button [PWRF] [ 3.049838] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.057453] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.076650] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.084671] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.100433] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.127949] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.154291] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.158391] Non-volatile memory driver v1.3 [ 3.160103] Linux agpgart interface v0.103 [ 3.192878] virtio_blk virtio1: [vda] 139528 512-byte logical blocks (71.4 MB/68.1 MiB) [ 3.195579] vda: detected capacity change from 0 to 71438336 [ 3.219358] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.222098] vdb: detected capacity change from 0 to 1073741824 [ 3.242143] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.245456] vdc: detected capacity change from 0 to 2621440000 [ 3.264178] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.266618] vdd: detected capacity change from 0 to 2621440000 [ 3.283677] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.286403] vde: detected capacity change from 0 to 4294967296 [ 3.301913] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.305601] vdf: detected capacity change from 0 to 4294967296 [ 3.313192] libphy: Fixed MDIO Bus: probed [ 3.331405] usbcore: registered new interface driver usbserial_generic [ 3.333368] usbserial: USB Serial support registered for generic [ 3.335272] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.338911] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.340417] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.342475] mousedev: PS/2 mouse device common for all mice [ 3.345375] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.347475] rtc_cmos 00:05: RTC can wake from S4 [ 3.352297] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.352545] rtc_cmos 00:05: registered as rtc0 [ 3.357332] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.361533] intel_pstate: CPU model not supported [ 3.364466] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.367133] hid: raw HID events driver (C) Jiri Kosina [ 3.370260] usbcore: registered new interface driver usbhid [ 3.372485] usbhid: USB HID core driver [ 3.374423] drop_monitor: Initializing network drop monitor service [ 3.377383] Initializing XFRM netlink socket [ 3.379428] NET: Registered protocol family 10 [ 3.381911] Segment Routing with IPv6 [ 3.382825] NET: Registered protocol family 17 [ 3.384322] mpls_gso: MPLS GSO support [ 3.391147] RAS: Correctable Errors collector initialized. [ 3.392499] AVX version of gcm_enc/dec engaged. [ 3.393440] AES CTR mode by8 optimization enabled [ 3.450629] sched_clock: Marking stable (3450606245, 0)->(4344499025, -893892780) [ 3.454261] registered taskstats version 1 [ 3.456163] Loading compiled-in X.509 certificates [ 3.457488] zswap: loaded using pool lzo/zbud [ 3.475737] Key type big_key registered [ 3.491341] Key type encrypted registered [ 3.492450] ima: No TPM chip found, activating TPM-bypass! [ 3.494371] ima: Allocated hash algorithm: sha1 [ 3.495800] ima: No architecture policies found [ 3.497584] evm: Initialising EVM extended attributes: [ 3.499596] evm: security.selinux [ 3.500731] evm: security.ima [ 3.501878] evm: security.capability [ 3.503434] evm: HMAC attrs: 0x1 [ 3.505584] rtc_cmos 00:05: setting system clock to 2026-06-09 12:48:15 UTC (1781009295) [ 3.511522] debug: unmapping init [mem 0xffffffffa9a03000-0xffffffffa9bfffff] [ 3.514196] debug: unmapping init [mem 0xffffffffa8782000-0xffffffffa8a58fff] [ 3.523540] Write protecting the kernel read-only data: 28672k [ 3.527561] debug: unmapping init [mem 0xffffffffa6e03000-0xffffffffa6ffffff] [ 3.530405] debug: unmapping init [mem 0xffffffffa7714000-0xffffffffa77fffff] [ 3.564203] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.572554] systemd[1]: Detected virtualization kvm. [ 3.574548] systemd[1]: Detected architecture x86-64. [ 3.576448] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.598668] systemd[1]: No hostname configured. [ 3.600358] systemd[1]: Set hostname to . [ 3.602414] random: systemd: uninitialized urandom read (16 bytes read) [ 3.604817] systemd[1]: Initializing machine ID from random generator. [ 3.748662] random: systemd: uninitialized urandom read (16 bytes read) [ 3.751869] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.757411] random: systemd: uninitialized urandom read (16 bytes read) [ 3.760198] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.765246] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.281179] device-mapper: uevent: version 1.0.3 [ 4.282654] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.913726] virtio_net virtio0 ens2: renamed from eth0 [ 4.916565] random: fast init done [ 4.974687] scsi host0: ata_piix [ 4.978997] scsi host1: ata_piix [ 4.980284] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.982577] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.158488] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.008584] random: crng init done [ 10.009465] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.213426] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.279843] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.513158] SELinux: Disabled at runtime. [ 11.576050] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.585042] systemd[1]: Detected virtualization kvm. [ 11.586304] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.033115] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.035482] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.041506] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.045265] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.048561] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.055507] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.062472] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ 12.205048] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.485039] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.755979] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.761851] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.901434] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.910321] EDAC sbridge: Ver: 1.1.2 [ 14.476249] Key type dns_resolver registered [ 14.784636] NFS: Registering the id_resolver key type [ 14.786599] Key type id_resolver registered [ 14.788438] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg139-server login: [ 49.440330] hrtimer: interrupt took 10009677 ns [ 56.146528] libcfs: loading out-of-tree module taints kernel. [ 56.197449] Key type ._llcrypt registered [ 56.204235] Key type .llcrypt registered [ 56.349179] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 75.491322] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 77.314728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 77.328967] alg: No test for adler32 (adler32-zlib) [ 79.007234] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 79.843316] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 81.559234] Key type lgssc registered [ 83.429078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 100.949884] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 134.565746] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 145.537698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.569043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 146.841376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 146.881140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 146.974561] Lustre: lustre-MDT0000: new disk, initializing [ 147.064686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.101449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 149.738988] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.702364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.802984] Lustre: 6354:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 160.841811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 160.846480] Lustre: Skipped 1 previous similar message [ 160.941690] Lustre: lustre-MDT0001: new disk, initializing [ 161.015700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 161.049539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 161.062607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 164.800336] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.172715] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 177.146837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 177.356591] Lustre: lustre-OST0000: new disk, initializing [ 177.359689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 177.364465] Lustre: 8255:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 177.422902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.603966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 179.621111] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 179.701884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 183.124792] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.092027] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 195.229678] Lustre: lustre-OST0001: new disk, initializing [ 195.234742] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 195.241780] Lustre: 9310:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 195.312646] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 200.281374] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.293287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 204.296366] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 204.312849] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 210.509428] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 221.225150] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 225.682231] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing check_logdir /tmp/testlogs/ [ 229.027763] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing yml_node [ 232.461719] Lustre: DEBUG MARKER: Client: 2.17.54.1 [ 234.356206] Lustre: DEBUG MARKER: MDS: 2.17.54.1 [ 236.224487] Lustre: DEBUG MARKER: OSS: 2.17.54.1 [ 237.320906] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 9 08:52:08 EDT 2026 [ 247.530673] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 248.581876] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 263.654716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 263.665387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 263.679840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.695901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.698841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.709451] Lustre: Skipped 1 previous similar message [ 265.715830] Lustre: Skipped 1 previous similar message [ 267.685614] Lustre: server umount lustre-MDT0000 complete [ 270.537665] LustreError: 6344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781009562 with bad export cookie 10188218507592629745 [ 270.539328] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 270.549839] LustreError: 6344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 270.816299] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.819554] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 270.822838] Lustre: Skipped 2 previous similar messages [ 270.829676] Lustre: Skipped 3 previous similar messages [ 275.938868] LustreError: 8262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 275.938956] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 275.950518] LustreError: 8262:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 275.963862] Lustre: Skipped 1 previous similar message [ 276.865864] Lustre: server umount lustre-MDT0001 complete [ 279.434933] Lustre: server umount lustre-OST0000 complete [ 282.007846] Lustre: server umount lustre-OST0001 complete [ 286.594907] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 293.812719] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 317.899901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 318.114909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 318.150804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 318.213727] Lustre: lustre-MDT0000: new disk, initializing [ 318.316329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 318.329444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 321.427411] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 329.341205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 329.408449] Lustre: 14674:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 329.444842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 329.447419] Lustre: Skipped 1 previous similar message [ 329.497705] Lustre: lustre-MDT0001: new disk, initializing [ 329.560042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 329.584971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 329.591439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 331.973455] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 337.343655] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 339.352211] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 343.817988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 343.991321] Lustre: 16416:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 345.371686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 345.456190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 348.068236] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 353.255191] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 356.041889] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 356.179189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 358.604729] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 358.741206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 360.928074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 360.930964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.936025] LustreError: Skipped 1 previous similar message [ 360.936932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 360.942876] Lustre: Skipped 1 previous similar message [ 371.171359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 371.179419] Lustre: Skipped 4 previous similar messages [ 374.239142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 374.365434] Lustre: server umount lustre-OST0000 complete [ 380.900394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 380.904361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 391.135200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 391.272283] Lustre: server umount lustre-MDT0000 complete [ 391.649799] LustreError: 15557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 391.655379] LustreError: 15557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 393.656152] LustreError: 14667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781009685 with bad export cookie 10188218507592631278 [ 393.659726] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 393.666091] LustreError: 14667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 393.876121] Lustre: server umount lustre-MDT0001 complete [ 397.935694] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:54:49 (1781009689) [ 409.545275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 409.794120] LustreError: 19146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 409.846732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.851723] Lustre: Skipped 1 previous similar message [ 412.988098] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 415.201767] LustreError: 19147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 418.530498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.163541] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 424.361949] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 426.331310] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 427.557048] Lustre: Failing over lustre-MDT0000 [ 427.690282] Lustre: server umount lustre-MDT0000 complete [ 428.583345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.586614] LustreError: 19141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 428.595284] Lustre: Skipped 2 previous similar messages [ 428.616107] LustreError: 19141:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 438.752063] LustreError: 19147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.757807] LustreError: 19147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 442.973066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.084663] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 443.233400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.238337] Lustre: Skipped 1 previous similar message [ 444.258297] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 444.261039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 444.279861] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 445.633369] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 450.193321] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 454.625561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 454.625798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.626918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 454.626926] Lustre: Skipped 5 previous similar messages [ 454.657615] Lustre: Skipped 1 previous similar message [ 458.991948] Lustre: server umount lustre-MDT0000 complete [ 459.743906] LustreError: 21002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 459.762706] LustreError: 21002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 461.147184] LustreError: 19128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781009753 with bad export cookie 10188218507592632692 [ 461.152728] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 461.153805] LustreError: 19128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 461.349266] Lustre: server umount lustre-MDT0001 complete [ 465.988683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.472780] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 473.396252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.825783] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 479.244380] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 481.604464] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 483.001634] Lustre: Failing over lustre-MDT0000 [ 483.147703] Lustre: server umount lustre-MDT0000 complete [ 483.808312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 495.074099] LustreError: 22135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.091235] LustreError: 22135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 498.040419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.094383] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.296323] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 499.299583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 499.302137] Lustre: Skipped 1 previous similar message [ 499.307896] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 500.353040] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 505.135171] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 509.921499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 509.923072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.927787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 509.927795] Lustre: Skipped 1 previous similar message [ 509.947235] Lustre: Skipped 3 previous similar messages [ 513.718483] Lustre: server umount lustre-MDT0000 complete [ 515.667907] LustreError: 22121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781009807 with bad export cookie 10188218507592633630 [ 515.671029] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 515.673962] LustreError: 22121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 515.813977] Lustre: server umount lustre-MDT0001 complete [ 520.286916] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:56:51 (1781009811) [ 528.177731] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 533.287725] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 553.054565] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 558.431067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.565348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 558.582919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 558.585957] Lustre: Skipped 1 previous similar message [ 558.626732] Lustre: lustre-MDT0000: new disk, initializing [ 558.628619] Lustre: Skipped 1 previous similar message [ 558.678064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 558.681075] Lustre: Skipped 3 previous similar messages [ 558.692912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 558.696661] Lustre: Skipped 1 previous similar message [ 561.040103] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 567.264947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.346787] Lustre: 28110:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 567.541587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 569.688692] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 573.814213] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 575.596731] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 578.774174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 578.902516] Lustre: 29850:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 580.469660] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 580.494987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 580.505827] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 581.813499] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 586.321805] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 590.417175] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 601.056751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 601.062767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.069119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 601.073466] Lustre: Skipped 2 previous similar messages [ 603.320847] Lustre: server umount lustre-OST0000 complete [ 611.034825] Lustre: server umount lustre-MDT0000 complete [ 611.295894] LustreError: 28121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.300964] LustreError: 28121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 612.392502] LustreError: 28888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781009904 with bad export cookie 10188218507592634036 [ 612.393904] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.396309] LustreError: 28888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 612.588645] Lustre: server umount lustre-MDT0001 complete [ 616.745814] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 617.925472] Key type lgssc unregistered [ 618.098679] LNet: 31674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.105014] LNetError: 31674:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 618.123125] LNet: Removed LNI 192.168.201.139@tcp [ 618.507216] Key type .llcrypt unregistered [ 618.508694] Key type ._llcrypt unregistered [ 628.657167] Key type ._llcrypt registered [ 628.658799] Key type .llcrypt registered [ 628.702204] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 629.118921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 629.149944] alg: No test for adler32 (adler32-zlib) [ 630.023637] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 630.126180] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 631.711155] Key type lgssc registered [ 632.139950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 635.256596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 635.262869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.451256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 638.092597] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 641.969255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.167492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 643.781209] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 645.858714] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 647.418964] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 650.305232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 650.472828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 652.530897] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 653.128288] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 655.574938] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 672.099136] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:59:23 (1781009963) [ 672.991522] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 673.866230] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 676.967154] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:59:28 (1781009968) [ 680.532497] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:59:32 (1781009972) [ 680.949609] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 684.746382] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 684.750791] Lustre: Skipped 2 previous similar messages [ 685.573815] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:59:37 (1781009977) [ 689.400929] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 689.403692] Lustre: Skipped 4 previous similar messages [ 696.735835] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:59:48 (1781009988) [ 697.907688] Lustre: *** cfs_fail_loc=131b, val=2*** [ 699.569594] Lustre: *** cfs_fail_loc=131b, val=2*** [ 703.582098] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:59:55 (1781009995) [ 704.415185] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 704.417267] Lustre: Skipped 9 previous similar messages [ 709.875108] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:00:01 (1781010001) [ 718.308588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 718.315068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.320321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 718.330113] Lustre: Skipped 1 previous similar message [ 718.337087] Lustre: Skipped 1 previous similar message [ 722.522250] Lustre: server umount lustre-OST0000 complete [ 724.448337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 724.452414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 724.458272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 728.545233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 728.545675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 730.550373] Lustre: server umount lustre-MDT0000 complete [ 732.727950] LustreError: 33026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010024 with bad export cookie 17318014203447768541 [ 732.729314] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 732.737136] LustreError: 33026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 733.022149] Lustre: server umount lustre-MDT0001 complete [ 739.305990] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 740.867351] Key type lgssc unregistered [ 741.085590] LNet: 40544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.089654] LNetError: 40544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.103601] LNet: Removed LNI 192.168.201.139@tcp [ 741.607956] Key type .llcrypt unregistered [ 741.609481] Key type ._llcrypt unregistered [ 752.413772] Key type ._llcrypt registered [ 752.415094] Key type .llcrypt registered [ 752.479918] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 760.964771] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 761.525000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 761.539881] alg: No test for adler32 (adler32-zlib) [ 762.457919] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 762.584488] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 764.199160] Key type lgssc registered [ 764.773484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 783.542515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 783.573881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 784.723621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 784.737650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 784.786167] Lustre: lustre-MDT0000: new disk, initializing [ 784.852680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 784.860293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 786.483975] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 792.384268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 792.444875] Lustre: 43652:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 792.480750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 792.483672] Lustre: Skipped 1 previous similar message [ 792.527647] Lustre: lustre-MDT0001: new disk, initializing [ 792.591959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 792.615510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 792.624479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 794.624601] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 798.673727] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 800.547866] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 804.285338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 804.448828] Lustre: lustre-OST0000: new disk, initializing [ 804.451430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 804.455840] Lustre: 45392:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 804.509354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.767617] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 811.538345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 811.543219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 811.598710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 812.295398] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 814.254000] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 814.340230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 816.316244] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 816.406777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 821.729537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 821.732302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 821.738138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 821.742432] Lustre: Skipped 1 previous similar message [ 823.463904] Lustre: server umount lustre-OST0000 complete [ 826.849806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.851738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 826.870534] Lustre: Skipped 1 previous similar message [ 826.872086] Lustre: Skipped 1 previous similar message [ 831.581888] Lustre: server umount lustre-MDT0000 complete [ 833.258220] LustreError: 44429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010125 with bad export cookie 142274390036942519 [ 833.258720] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 833.266811] LustreError: 44429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 833.453541] Lustre: server umount lustre-MDT0001 complete [ 838.037654] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:02:09 (1781010129) [ 841.020715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 841.257519] LustreError: 47698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 841.264172] LustreError: 47698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 841.288186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 843.147735] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 846.306268] LustreError: 47699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 847.320969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.470579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 849.783238] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 852.343985] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 853.748742] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 856.513780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 856.644077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 858.840182] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 860.984564] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 873.432492] LustreError: 49908:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 873.782255] Lustre: Modifying parameter general.timeout=1 in log params [ 874.849627] Lustre: Modifying parameter general.timeout=4 in log params [ 874.854853] Lustre: Skipped 2 previous similar messages [ 877.188826] Lustre: Modifying parameter general.timeout=10 in log params [ 877.191102] Lustre: Skipped 5 previous similar messages [ 881.365302] Lustre: Modifying parameter general.timeout=22 in log params [ 881.367934] Lustre: Skipped 11 previous similar messages [ 889.541977] Lustre: Modifying parameter general.timeout=43 in log params [ 889.544519] Lustre: Skipped 20 previous similar messages [ 905.847730] Lustre: Modifying parameter general.timeout=84 in log params [ 905.850262] Lustre: Skipped 40 previous similar messages [ 926.790841] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:03:38 (1781010218) [ 927.930705] Lustre: 56818:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 929.487215] Lustre: 57015:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 929.492392] Lustre: 57015:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 931.614305] Lustre: 57310:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 931.620566] Lustre: 57310:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 2 previous similar messages [ 938.282101] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 938.285704] Lustre: Skipped 64 previous similar messages [ 941.690780] Lustre: 58661:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 941.698699] LustreError: 58661:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 943.634553] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:03:55 (1781010235) [ 949.728675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 949.733398] LustreError: Skipped 1 previous similar message [ 949.736647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.744368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 949.747016] Lustre: Skipped 1 previous similar message [ 955.165621] Lustre: server umount lustre-MDT0000 complete [ 956.386461] LustreError: 47698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.393349] LustreError: 47698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 956.903715] LustreError: 47679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010248 with bad export cookie 142274390036943387 [ 956.907677] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 956.908856] LustreError: 47679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 957.057646] Lustre: server umount lustre-MDT0001 complete [ 969.093296] Lustre: server umount lustre-OST0000 complete [ 981.535694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.655328] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 981.796895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.447916] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 987.087733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.158419] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 987.174966] Lustre: 61191:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 987.219351] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 987.221591] Lustre: Skipped 3 previous similar messages [ 989.223470] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 991.497583] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 992.982714] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 995.656885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 995.752123] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 998.675254] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1001.176156] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1002.485628] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1002.489597] Lustre: Skipped 6 previous similar messages [ 1004.513561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.513812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1004.518315] Lustre: Skipped 2 previous similar messages [ 1004.524339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1006.181388] Lustre: server umount lustre-OST0000 complete [ 1008.098312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1008.102961] LustreError: Skipped 1 previous similar message [ 1009.632822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.638395] Lustre: Skipped 2 previous similar messages [ 1012.086386] Lustre: server umount lustre-MDT0000 complete [ 1013.574710] LustreError: 61913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010305 with bad export cookie 142274390036945781 [ 1013.575223] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1013.580152] LustreError: 61913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1013.713534] Lustre: server umount lustre-MDT0001 complete [ 1018.056663] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1019.330030] Key type lgssc unregistered [ 1019.473650] LNet: 64599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.479149] LNetError: 64599:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.488157] LNet: Removed LNI 192.168.201.139@tcp [ 1019.852510] Key type .llcrypt unregistered [ 1019.853644] Key type ._llcrypt unregistered [ 1031.275990] Key type ._llcrypt registered [ 1031.278373] Key type .llcrypt registered [ 1031.330667] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1031.724484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1031.763032] alg: No test for adler32 (adler32-zlib) [ 1032.676590] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 1032.769175] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1034.386436] Key type lgssc registered [ 1034.857225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1037.944359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1037.951367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.232957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.887958] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1044.572309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.835031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1046.689974] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1048.933366] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1050.461387] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1053.386057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1053.620161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1056.063495] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1058.345827] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1072.186855] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1075.679924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1075.683047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.688657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1080.290109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1080.290921] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.693151] Lustre: server umount lustre-MDT0000 complete [ 1083.191111] LustreError: 65954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010375 with bad export cookie 12595949488954409237 [ 1083.193690] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1083.196960] LustreError: 65954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1083.361117] Lustre: server umount lustre-MDT0001 complete [ 1094.176753] Lustre: server umount lustre-OST0000 complete [ 1107.392296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.541214] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1107.556782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1107.718095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.487560] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1113.230899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1113.303808] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1113.316925] Lustre: 70519:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1113.507054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1115.138896] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1117.834083] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1119.560153] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1122.498839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1122.588783] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1122.750393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.041922] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1127.254349] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1127.973775] Lustre: 72756:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1128.766805] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1130.979827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1130.980760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1130.981493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1130.981500] Lustre: Skipped 2 previous similar messages [ 1130.993715] Lustre: Skipped 2 previous similar messages [ 1132.317184] Lustre: server umount lustre-OST0000 complete [ 1134.047829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1134.050810] LustreError: Skipped 1 previous similar message [ 1134.052599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.061356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1134.063845] Lustre: Skipped 1 previous similar message [ 1135.927974] Lustre: server umount lustre-MDT0000 complete [ 1136.099399] LustreError: 72575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1136.107359] LustreError: 72575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1137.743902] LustreError: 70511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010429 with bad export cookie 12595949488954410049 [ 1137.746558] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1137.751448] LustreError: 70511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1137.908718] Lustre: server umount lustre-MDT0001 complete [ 1143.131753] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1144.510841] Key type lgssc unregistered [ 1144.659322] LNet: 73942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1144.666644] LNetError: 73942:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1144.673802] LNet: Removed LNI 192.168.201.139@tcp [ 1145.073955] Key type .llcrypt unregistered [ 1145.075366] Key type ._llcrypt unregistered [ 1156.127110] Key type ._llcrypt registered [ 1156.128478] Key type .llcrypt registered [ 1156.174433] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1156.522544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1156.538638] alg: No test for adler32 (adler32-zlib) [ 1157.407949] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 1157.523721] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1159.143139] Key type lgssc registered [ 1159.635238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1162.882177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1162.891505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.124472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1165.798365] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1169.520329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1169.786376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1171.403645] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1174.064800] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1175.664954] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1178.329919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1178.617259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.089550] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1183.496259] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1192.268065] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1193.064950] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1193.069502] Lustre: Skipped 1 previous similar message [ 1194.188285] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1194.190491] Lustre: Skipped 2 previous similar messages [ 1196.532535] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1196.534356] Lustre: Skipped 5 previous similar messages [ 1200.849202] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1200.853093] Lustre: Skipped 9 previous similar messages [ 1208.869159] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1208.873314] Lustre: Skipped 18 previous similar messages [ 1224.883141] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1224.886258] Lustre: Skipped 42 previous similar messages [ 1234.429430] Lustre: 83042:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1235.475688] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:08:47 (1781010527) [ 1236.064772] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1236.851070] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:08:48 (1781010528) [ 1241.057846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1241.057946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.059336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.062157] LustreError: Skipped 1 previous similar message [ 1241.069379] Lustre: Skipped 1 previous similar message [ 1246.176981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1246.179293] Lustre: Skipped 2 previous similar messages [ 1251.300815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1251.304656] Lustre: Skipped 1 previous similar message [ 1252.319222] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1252.397357] Lustre: server umount lustre-OST0000 complete [ 1256.418193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1256.419585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1256.426536] Lustre: Skipped 1 previous similar message [ 1261.536376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.539015] Lustre: Skipped 2 previous similar messages [ 1268.191157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1268.269225] Lustre: server umount lustre-MDT0000 complete [ 1269.766664] LustreError: 75294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010561 with bad export cookie 1162834813532244273 [ 1269.769929] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1269.770809] LustreError: 75294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1269.906739] Lustre: server umount lustre-MDT0001 complete [ 1273.750464] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1274.926385] Key type lgssc unregistered [ 1275.086547] LNet: 84276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1275.090420] LNetError: 84276:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1275.104371] LNet: Removed LNI 192.168.201.139@tcp [ 1275.479572] Key type .llcrypt unregistered [ 1275.481115] Key type ._llcrypt unregistered [ 1276.980273] Key type ._llcrypt registered [ 1276.981918] Key type .llcrypt registered [ 1277.028308] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1278.753237] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules [ 1279.055254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1279.107994] alg: No test for adler32 (adler32-zlib) [ 1279.981430] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 1280.072085] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1281.663152] Key type lgssc registered [ 1282.110835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1282.599268] LustreError: 85394:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1290.838403] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1294.524018] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1303.896959] LustreError: 85394:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1303.915501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1305.165945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.663981] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1310.210017] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:10:01 (1781010601) [ 1311.976661] LustreError: 86432:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1311.982567] LustreError: 86432:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1311.986613] LustreError: 86432:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1312.108119] Lustre: server umount lustre-MDT0000 complete [ 1317.179889] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1318.425930] Key type lgssc unregistered [ 1318.587364] LNet: 87888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1318.590800] LNetError: 87888:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1318.597352] LNet: Removed LNI 192.168.201.139@tcp [ 1318.906571] Key type .llcrypt unregistered [ 1318.907829] Key type ._llcrypt unregistered [ 1325.892788] Key type ._llcrypt registered [ 1325.893816] Key type .llcrypt registered [ 1325.935501] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1326.267144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1326.315344] alg: No test for adler32 (adler32-zlib) [ 1327.171948] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 1327.266349] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1328.855176] Key type lgssc registered [ 1329.242715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1332.199189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1332.206197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.405038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.832873] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1338.126516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.383686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1339.776129] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1341.628475] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1343.040221] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1345.477357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1345.713182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1347.713952] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1349.592022] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1382.900075] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:11:14 (1781010674) [ 1383.578220] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1384.296996] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:11:15 (1781010675) [ 1389.536441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1389.539410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.544935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.048433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.050250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1391.527375] Lustre: server umount lustre-MDT0000 complete [ 1392.975840] LustreError: 89791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010684 with bad export cookie 18319294753907787994 [ 1392.979711] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1392.980747] LustreError: 89791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1393.093764] Lustre: server umount lustre-MDT0001 complete [ 1404.989454] Lustre: server umount lustre-OST0000 complete [ 1409.169984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1409.361356] LustreError: 93015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1409.367772] LustreError: 93015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1409.415532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1410.891600] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1413.984876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.243527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1415.659252] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1417.615084] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1419.006893] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1423.880596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1423.951145] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1423.954640] LustreError: 93005:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1423.965326] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1423.968844] LustreError: 94870:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1423.972104] LustreError: 94870:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1423.974728] LustreError: 94870:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1424.005338] Lustre: server umount lustre-OST0000 complete [ 1424.007140] LustreError: 94870:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1426.978194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1427.047605] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1427.050606] LustreError: 93005:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1427.054264] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1427.057545] LustreError: 95335:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1427.059426] LustreError: 95335:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1427.061226] LustreError: 95335:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1427.083671] Lustre: server umount lustre-OST0000 complete [ 1427.085568] LustreError: 95335:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1430.337891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1430.404733] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1430.406291] Lustre: Found index 0 for lustre-OST0000, updating log [ 1430.408608] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1430.434662] Lustre: lustre-OST0000: new disk, initializing [ 1430.437046] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1430.440857] Lustre: 95862:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1430.547057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1432.736193] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1434.197740] Lustre: Failing over lustre-OST0000 [ 1434.251498] Lustre: server umount lustre-OST0000 complete [ 1439.711981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1439.716412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1439.724309] Lustre: Skipped 1 previous similar message [ 1439.727689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1439.731125] Lustre: Skipped 2 previous similar messages [ 1441.818635] Lustre: server umount lustre-MDT0000 complete [ 1443.133276] LustreError: 92995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010735 with bad export cookie 18319294753907790724 [ 1443.134870] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1443.137362] LustreError: 92995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1443.252943] Lustre: server umount lustre-MDT0001 complete [ 1446.218745] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:12:17 (1781010737) [ 1451.973285] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1455.997382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.167101] LustreError: 98168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.208396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1457.681351] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1461.196511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.216733] LustreError: 98169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1462.998427] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1464.221732] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1467.073359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1469.592405] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1473.073093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1473.106711] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1473.148906] Lustre: lustre-OST0001: new disk, initializing [ 1473.150879] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1473.154350] Lustre: 100431:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1475.359798] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1477.637928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1477.641486] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1477.681492] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1483.950311] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1493.038122] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1496.670773] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.39@tcp (stopping) [ 1496.673522] Lustre: Skipped 2 previous similar messages [ 1497.059748] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.067120] Lustre: Skipped 3 previous similar messages [ 1500.706627] Lustre: server umount lustre-MDT0001 complete [ 1501.781898] LustreError: 100441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1504.213788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1504.257972] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1504.260389] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1504.280966] Lustre: 98157:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1504.573255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1504.575870] Lustre: Skipped 3 previous similar messages [ 1505.646541] LustreError: 102088:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1505.651717] LustreError: 102088:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1505.655294] LustreError: 98160:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1505.692354] LustreError: 102098:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.139@tcp: cfg command failed: rc = -17 [ 1505.697218] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1505.697218] [ 1506.069637] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1509.722994] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:13:21 (1781010801) [ 1509.863288] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1516.236929] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1522.634501] Lustre: 103989:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1532.129972] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1533.556391] Lustre: 105123:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1586.656315] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1586.657086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1586.657583] Lustre: Skipped 5 previous similar messages [ 1586.667824] Lustre: Skipped 3 previous similar messages [ 1591.873744] Lustre: server umount lustre-MDT0000 complete [ 1593.667063] LustreError: 100430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781010885 with bad export cookie 18319294753907791564 [ 1593.667650] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1593.671673] LustreError: 100430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1593.836911] Lustre: server umount lustre-MDT0001 complete [ 1605.135387] Lustre: server umount lustre-OST0000 complete [ 1616.813973] Lustre: server umount lustre-OST0001 complete [ 1618.924406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1622.190854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1625.134050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1628.609701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1635.175759] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1638.978742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.987239] Lustre: lustre-MDT0000: reset Object Index mappings [ 1639.163630] LustreError: 109046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1639.169250] LustreError: 109046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1639.194509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1640.713450] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1644.008197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.016143] Lustre: lustre-MDT0001: reset Object Index mappings [ 1645.623222] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1646.612040] Lustre: 110161:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1648.926197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1650.276634] LustreError: 110515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.279942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1651.209338] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1654.335483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1654.374721] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1656.655343] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1659.842469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1659.881900] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 1659.882787] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 1661.084021] Lustre: 112002:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1676.711922] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:16:08 (1781010968) [ 1711.074479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1711.075376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.083819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1711.086264] Lustre: Skipped 7 previous similar messages [ 1712.374258] Lustre: server umount lustre-MDT0000 complete [ 1713.810808] LustreError: 109026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781011005 with bad export cookie 18319294753907838331 [ 1713.814900] LustreError: 109026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1713.816954] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1716.193794] LustreError: 112240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.198689] LustreError: 112240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1720.127553] Lustre: server umount lustre-MDT0001 complete [ 1727.702860] Lustre: server umount lustre-OST0000 complete [ 1735.454908] Lustre: server umount lustre-OST0001 complete [ 1737.741803] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 1741.322723] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1754.426962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.531870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1754.547039] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1754.585296] Lustre: lustre-MDT0000: new disk, initializing [ 1754.633735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1756.145315] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1759.300280] Lustre: server umount lustre-MDT0000 complete [ 1762.809661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1762.928371] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1762.938099] Lustre: 116207:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1762.945505] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1764.539356] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1766.476390] Lustre: server umount lustre-MDT0000 complete [ 1769.764367] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:17:41 (1781011061) [ 1770.321755] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1771.009494] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:17:42 (1781011062) [ 1777.652207] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1781.704187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1781.918795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1781.922042] Lustre: Skipped 5 previous similar messages [ 1783.409505] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1786.745202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.790658] Lustre: 118225:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1786.822117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1786.824989] Lustre: Skipped 1 previous similar message [ 1786.863066] Lustre: lustre-MDT0001: new disk, initializing [ 1786.915241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1786.919946] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1788.295740] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1790.765309] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1793.349670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1793.457416] Lustre: lustre-OST0000: new disk, initializing [ 1793.458978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1793.461799] Lustre: 119770:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1793.947070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1793.950807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1794.003261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1795.612796] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1800.044304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1800.104576] Lustre: 120622:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1801.486952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1801.492843] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1801.527917] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1802.085046] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1806.680381] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1807.960571] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1930.679674] Lustre: 126976:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1930.682351] Lustre: 126976:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1930.684288] Lustre: 126976:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1930.687479] Lustre: 126976:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1930.690134] Lustre: 126976:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1930.693120] Lustre: 126976:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2019.960207] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:21:51 (1781011311) [ 2031.368521] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 2031.370475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.373289] Lustre: Skipped 5 previous similar messages [ 2031.375217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.379012] Lustre: Skipped 4 previous similar messages [ 2037.634655] Lustre: server umount lustre-MDT0000 complete [ 2038.875324] LustreError: 118219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781011330 with bad export cookie 18319294753907891895 [ 2038.878564] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2038.878981] LustreError: 118219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2042.335934] LustreError: 120464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.340601] LustreError: 120464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2045.067240] Lustre: server umount lustre-MDT0001 complete [ 2052.353810] Lustre: server umount lustre-OST0000 complete [ 2059.671768] Lustre: server umount lustre-OST0001 complete [ 2061.636756] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 2064.857491] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2076.813269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2076.903329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2076.920202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2076.922212] Lustre: Skipped 1 previous similar message [ 2076.959081] Lustre: lustre-MDT0000: new disk, initializing [ 2076.960676] Lustre: Skipped 1 previous similar message [ 2076.983701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.986236] Lustre: Skipped 3 previous similar messages [ 2076.993588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2078.292733] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2082.744706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.797493] Lustre: 129958:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2082.902073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2084.304751] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2087.444178] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2088.609324] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2090.835085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2090.932688] Lustre: 131699:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2092.536649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2092.571473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2092.792227] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2095.823451] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2098.911491] Lustre: lustre-MDD0000: changelog on [ 2099.829994] Lustre: lustre-MDD0001: changelog on [ 2984.681784] Lustre: lustre-MDD0001: changelog off [ 2985.725587] Lustre: 132496:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2986.207779] Lustre: lustre-MDD0000: changelog off [ 2993.633625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2993.633919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2993.637876] Lustre: Skipped 5 previous similar messages [ 2993.644646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2993.646805] Lustre: Skipped 13 previous similar messages [ 2994.828758] Lustre: server umount lustre-OST0000 complete [ 2998.755263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.763756] Lustre: Skipped 1 previous similar message [ 2998.767214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3002.916554] Lustre: server umount lustre-MDT0000 complete [ 3003.874733] LustreError: 129970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3003.882395] LustreError: 129970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3004.964655] LustreError: 129950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781012296 with bad export cookie 18319294753908700507 [ 3004.967377] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3004.972283] LustreError: 129950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3005.178110] Lustre: server umount lustre-MDT0001 complete [ 3010.168302] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 3011.452456] Key type lgssc unregistered [ 3011.595566] LNet: 140405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3011.599252] LNetError: 140405:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3011.618251] LNet: Removed LNI 192.168.201.139@tcp [ 3012.040534] Key type .llcrypt unregistered [ 3012.042279] Key type ._llcrypt unregistered [ 3021.992726] Key type ._llcrypt registered [ 3021.994922] Key type .llcrypt registered [ 3022.044400] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 3030.654727] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3031.202139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3031.236698] alg: No test for adler32 (adler32-zlib) [ 3032.132954] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 3032.238800] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 3033.839141] Key type lgssc registered [ 3034.399367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3052.420664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3052.441675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.561752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3053.579479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3053.620405] Lustre: lustre-MDT0000: new disk, initializing [ 3053.667754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3053.679474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3055.873845] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3061.627160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3061.712964] Lustre: 143546:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3061.747130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3061.753724] Lustre: Skipped 1 previous similar message [ 3061.793345] Lustre: lustre-MDT0001: new disk, initializing [ 3061.834358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3061.863276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3061.867271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3063.585745] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3067.536744] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3069.193396] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3072.431028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3072.577900] Lustre: lustre-OST0000: new disk, initializing [ 3072.581042] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3072.584364] Lustre: 145290:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3072.629216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.359706] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3075.654874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3075.660780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3075.711956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3079.402317] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3083.209941] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_module obdecho/obdecho [ 3084.233306] LustreError: 146163:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3084.238483] LustreError: 146163:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3085.198427] LustreError: 146259:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 3091.424798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.431920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3091.439267] Lustre: Skipped 2 previous similar messages [ 3091.445134] Lustre: Skipped 2 previous similar messages [ 3096.313800] Lustre: server umount lustre-MDT0000 complete [ 3098.065724] LustreError: 145289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781012390 with bad export cookie 3667086361857910192 [ 3098.066242] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3098.070896] LustreError: 145289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3098.214235] Lustre: server umount lustre-MDT0001 complete [ 3110.462395] Lustre: server umount lustre-OST0000 complete [ 3123.821318] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 3130.725449] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3152.745202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3152.903398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3152.928706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3152.995164] Lustre: lustre-MDT0000: new disk, initializing [ 3153.071039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3153.082271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3155.312610] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3161.794336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.881809] Lustre: 149935:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3161.925314] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3161.929511] Lustre: Skipped 1 previous similar message [ 3161.932792] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3161.977163] Lustre: lustre-MDT0001: new disk, initializing [ 3162.031908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3162.050083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3162.059365] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3164.088420] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3168.437421] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3170.476720] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3174.170467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3174.318582] Lustre: lustre-OST0000: new disk, initializing [ 3174.321061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3174.324748] Lustre: 151678:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3174.384237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3174.452892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3174.462440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3174.544962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3177.136192] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3181.252166] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3183.364531] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3183.467530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3185.763875] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3185.901254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3188.196560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.203637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3188.210070] Lustre: Skipped 1 previous similar message [ 3193.187740] Lustre: server umount lustre-OST0000 complete [ 3197.921779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3197.930347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.937976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3197.940463] Lustre: Skipped 1 previous similar message [ 3201.321332] Lustre: server umount lustre-MDT0000 complete [ 3203.332743] LustreError: 149927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781012495 with bad export cookie 3667086361857911151 [ 3203.337943] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3203.338733] LustreError: 149927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3203.505912] Lustre: server umount lustre-MDT0001 complete [ 3205.488719] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:41:36 (1781012496) [ 3206.341736] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3207.279464] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 09:41:38 (1781012498) [ 3211.347320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3211.609573] LustreError: 153937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3211.625932] LustreError: 153937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3211.653507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.531886] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3216.864091] LustreError: 153938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3217.356095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.118140] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3221.600405] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3223.402717] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3227.141230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3230.527754] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3233.494437] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3242.977671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3242.986452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3242.990959] Lustre: Skipped 1 previous similar message [ 3242.993347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3242.997704] Lustre: Skipped 2 previous similar messages [ 3247.073823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3248.412250] Lustre: server umount lustre-MDT0000 complete [ 3249.240374] LustreError: 153934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.825961] LustreError: 155552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781012542 with bad export cookie 3667086361857912033 [ 3250.830517] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3250.847590] LustreError: 155552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3252.195192] LustreError: 153932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.196596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3252.208190] Lustre: Skipped 4 previous similar messages [ 3257.047043] Lustre: server umount lustre-MDT0001 complete [ 3260.174680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3267.473730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3267.593830] LustreError: 157437:0:(llog.c:1646:llog_backup()) MGC192.168.201.139@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3267.598734] Lustre: 157437:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.139@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3273.877410] LustreError: 157442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3273.894137] LustreError: 157442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3284.055068] LustreError: 157443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.074475] LustreError: 157443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3300.320314] LustreError: 157460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3300.333695] LustreError: 157460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 3328.991276] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3328.996759] Lustre: 157437:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.139@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3335.255107] LustreError: 157444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3335.264113] LustreError: 157444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 3369.456208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3369.489440] Lustre: Skipped 2 previous similar messages [ 3370.539225] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3374.377451] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3384.630238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3395.059781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 3400.261652] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3400.677400] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3407.457975] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3409.966507] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3416.234316] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3418.756038] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3441.635900] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3441.644372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3441.646455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3441.646463] Lustre: Skipped 2 previous similar messages [ 3441.647039] Lustre: Skipped 3 previous similar messages [ 3446.753795] LustreError: 157442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.757944] Lustre: server umount lustre-MDT0000 complete [ 3446.760212] LustreError: 157442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 47 previous similar messages [ 3450.252465] LustreError: 153919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781012742 with bad export cookie 3667086361857918480 [ 3450.260035] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3450.263986] LustreError: 153919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3450.686633] Lustre: server umount lustre-MDT0001 complete [ 3464.220234] Lustre: server umount lustre-OST0000 complete [ 3482.738781] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 3491.826815] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3523.577178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.768760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3523.788554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3523.850918] Lustre: lustre-MDT0000: new disk, initializing [ 3523.933373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3523.936592] Lustre: Skipped 1 previous similar message [ 3523.948685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3527.883582] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3537.335933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3537.434968] Lustre: 162282:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3537.617367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3541.141776] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3547.697445] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3551.319093] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3557.434995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3557.719706] Lustre: lustre-OST0000: new disk, initializing [ 3557.722039] Lustre: Skipped 1 previous similar message [ 3557.725624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3557.730980] Lustre: Skipped 2 previous similar messages [ 3557.735549] Lustre: 164024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3562.610048] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3565.104812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3565.111716] Lustre: Skipped 1 previous similar message [ 3565.118026] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3565.244642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3568.502631] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3571.998411] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3572.158821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3576.441583] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3576.778548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3580.896126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3580.900531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.905303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3580.928546] Lustre: Skipped 2 previous similar messages [ 3580.943172] Lustre: Skipped 3 previous similar messages [ 3584.612391] Lustre: server umount lustre-OST0000 complete [ 3589.089329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3589.101495] LustreError: Skipped 1 previous similar message [ 3602.399236] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3602.609845] Lustre: server umount lustre-MDT0000 complete [ 3606.496494] LustreError: 162289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3606.507984] LustreError: 162289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3606.547326] LustreError: 163070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781012898 with bad export cookie 3667086361857935301 [ 3606.558624] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3606.565331] LustreError: 163070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3606.896646] Lustre: server umount lustre-MDT0001 complete [ 3615.274533] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 09:48:26 (1781012906) [ 3621.039496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3625.089537] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3633.036472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3637.686191] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3643.582505] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3647.026753] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3653.518154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3653.852290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.855174] Lustre: Skipped 4 previous similar messages [ 3659.741046] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3664.408342] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3679.263799] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 09:49:29 (1781012969) [ 3684.834669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3684.839989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3684.849204] Lustre: Skipped 2 previous similar messages [ 3684.854563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3684.859792] Lustre: Skipped 7 previous similar messages [ 3689.222869] Lustre: server umount lustre-OST0000 complete [ 3707.361044] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3707.539365] Lustre: server umount lustre-MDT0000 complete [ 3711.257465] LustreError: 166269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781013003 with bad export cookie 3667086361857936183 [ 3711.259120] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3711.266824] LustreError: 166269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3711.517689] Lustre: server umount lustre-MDT0001 complete [ 3722.125786] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 3724.594241] Key type lgssc unregistered [ 3724.868878] LNet: 169659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3724.875585] LNetError: 169659:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3724.901160] LNet: Removed LNI 192.168.201.139@tcp [ 3725.583387] Key type .llcrypt unregistered [ 3725.585868] Key type ._llcrypt unregistered [ 3744.545431] Key type ._llcrypt registered [ 3744.547534] Key type .llcrypt registered [ 3744.631401] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3745.696209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3745.858756] alg: No test for adler32 (adler32-zlib) [ 3747.010309] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 3747.291689] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 3749.015356] Key type lgssc registered [ 3750.254741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3757.462939] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3757.476499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3773.023366] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3778.143355] LustreError: 170885:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.139@tcp: failed processing log, type 4: rc = -110 [ 3783.263441] LustreError: 170885:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3783.287274] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3783.311501] LustreError: MGC192.168.201.139@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3783.343714] LustreError: 170885:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3783.362251] LustreError: 170885:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 3783.372198] LustreError: 170885:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 3783.495570] Lustre: server umount lustre-OST0000 complete [ 3783.498213] LustreError: 170885:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3790.583519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3792.059065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.384630] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3802.597125] LustreError: 171384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3802.624451] LustreError: 171384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3804.590490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3804.974538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3808.886408] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3814.028796] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3817.539271] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3823.915996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3824.079383] LustreError: 172971:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3824.086381] Lustre: 172971:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.139@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3824.193248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.649953] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3834.551992] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3839.978108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.982579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.994120] Lustre: Skipped 1 previous similar message [ 3840.014572] Lustre: Skipped 2 previous similar messages [ 3842.555948] Lustre: server umount lustre-MDT0000 complete [ 3845.090728] LustreError: 171385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3845.125755] LustreError: 171385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3847.055034] LustreError: 171364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781013139 with bad export cookie 11628553552394464898 [ 3847.075065] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3847.077799] LustreError: 171364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3847.595051] Lustre: server umount lustre-MDT0001 complete [ 3865.042820] Lustre: server umount lustre-OST0000 complete [ 3880.252778] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 3898.349937] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3946.945523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3947.235628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3947.254855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3947.295991] Lustre: lustre-MDT0000: new disk, initializing [ 3947.462447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3947.505327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3952.855461] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3965.289996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3965.403351] Lustre: 176375:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3965.450920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3965.463184] Lustre: Skipped 1 previous similar message [ 3965.564225] Lustre: lustre-MDT0001: new disk, initializing [ 3965.739221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3965.803664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3965.820485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3973.005986] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3983.242619] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3986.648662] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3994.361615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3994.994620] Lustre: lustre-OST0000: new disk, initializing [ 3995.015806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3995.037644] Lustre: 178115:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3995.187485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.124874] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4002.388037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4002.402881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4002.532060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4010.714803] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4016.604852] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4016.887547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4022.669275] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4022.942518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4028.385511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4028.396560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4028.404842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4028.425710] Lustre: Skipped 2 previous similar messages [ 4028.438562] Lustre: Skipped 1 previous similar message [ 4031.916799] Lustre: server umount lustre-OST0000 complete [ 4037.601680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4037.628089] LustreError: Skipped 1 previous similar message [ 4037.645896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.681145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4043.362504] Lustre: server umount lustre-MDT0000 complete [ 4043.744629] LustreError: 178123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.777301] LustreError: 178123:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4048.879407] LustreError: 176381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4048.901753] LustreError: 176381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4048.969386] LustreError: 176366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781013340 with bad export cookie 11628553552394465612 [ 4048.995856] LustreError: 176366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4048.998058] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4049.779392] Lustre: server umount lustre-MDT0001 complete [ 4063.354550] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 09:55:53 (1781013353) [ 4085.552322] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 4089.920744] Key type lgssc unregistered [ 4090.333074] LNet: 180898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4090.355488] LNetError: 180898:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4090.390633] LNet: Removed LNI 192.168.201.139@tcp [ 4091.568297] Key type .llcrypt unregistered [ 4091.576462] Key type ._llcrypt unregistered [ 4115.462325] Key type ._llcrypt registered [ 4115.465130] Key type .llcrypt registered [ 4115.552197] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 4116.963540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4117.021726] alg: No test for adler32 (adler32-zlib) [ 4118.138832] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 4118.418684] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 4120.097388] Key type lgssc registered [ 4121.805984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4131.433028] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4131.483588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.383904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4139.398967] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4143.596274] LustreError: 182102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.622479] LustreError: 182102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4148.712481] LustreError: 182103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4150.363114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4150.954086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4157.148990] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4164.822279] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4170.049228] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4180.993470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4181.297075] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4181.304605] LustreError: 183741:0:(llog.c:1687:llog_backup()) MGC192.168.201.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4181.320967] Lustre: 183741:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.139@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4181.519316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4190.235873] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4197.532119] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4200.930564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.934700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4200.959882] Lustre: Skipped 1 previous similar message [ 4206.057615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4206.089454] Lustre: Skipped 4 previous similar messages [ 4206.646692] Lustre: server umount lustre-MDT0000 complete [ 4211.186142] LustreError: 182097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4211.219866] LustreError: 182097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4211.449029] LustreError: 182083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781013503 with bad export cookie 7329159270329410310 [ 4211.467336] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4211.472280] LustreError: 182083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4211.943865] Lustre: server umount lustre-MDT0001 complete [ 4226.668707] Lustre: server umount lustre-OST0000 complete [ 4238.907272] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 4255.751733] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 4306.754800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.346178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4307.472326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4307.673492] Lustre: lustre-MDT0000: new disk, initializing [ 4307.850044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4307.922178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4313.477624] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4326.514338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4326.659412] Lustre: 187147:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4326.700806] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4326.703309] Lustre: Skipped 1 previous similar message [ 4326.705903] LustreError: 187911:0:(llog.c:1687:llog_backup()) MGC192.168.201.139@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4326.711325] LustreError: 187911:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 4326.715769] Lustre: 187911:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.139@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4326.734165] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4326.737836] Lustre: Skipped 1 previous similar message [ 4326.811531] Lustre: lustre-MDT0001: new disk, initializing [ 4326.956974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4326.993269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4327.018364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4332.951391] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4341.572231] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4345.886942] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4353.777731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4354.160803] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4354.165082] Lustre: Skipped 1 previous similar message [ 4354.169160] LustreError: 188863:0:(llog.c:1687:llog_backup()) MGC192.168.201.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4354.176274] LustreError: 188863:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 4354.182722] Lustre: 188863:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.139@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4354.202629] Lustre: 188863:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4354.237155] Lustre: lustre-OST0000: new disk, initializing [ 4354.246716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4354.261450] Lustre: 188888:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4354.419777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4361.472376] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4361.948648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4361.970144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4362.203024] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4372.008167] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4378.685553] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4379.108338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4384.623245] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4384.900866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4387.825013] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4387.830301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4387.838252] LustreError: Skipped 1 previous similar message [ 4387.842136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4387.880442] Lustre: Skipped 2 previous similar messages [ 4392.934159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4392.939303] Lustre: Skipped 2 previous similar messages [ 4393.946244] Lustre: server umount lustre-OST0000 complete [ 4398.056910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.061510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4398.073548] Lustre: Skipped 1 previous similar message [ 4398.085766] Lustre: Skipped 1 previous similar message [ 4408.290232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4408.308874] Lustre: Skipped 3 previous similar messages [ 4412.383565] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4412.675197] Lustre: server umount lustre-MDT0000 complete [ 4413.415676] LustreError: 187154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.454723] LustreError: 187154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4417.799369] LustreError: 187137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781013709 with bad export cookie 7329159270329411024 [ 4417.805276] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4417.816840] LustreError: 187137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4418.786660] Lustre: server umount lustre-MDT0001 complete [ 4429.940343] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:02:00 (1781013720) [ 4451.024242] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 4464.739293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4465.122074] LustreError: 191867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4465.184559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4470.507688] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4475.363684] LustreError: 191868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.391418] LustreError: 191868:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4480.106896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4485.764945] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4488.983068] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4495.883491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4502.705349] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4511.626916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4511.916359] Lustre: lustre-OST0001: new disk, initializing [ 4511.925769] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4511.935080] Lustre: 194124:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4512.083884] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4512.093530] Lustre: Skipped 2 previous similar messages [ 4518.518787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4518.529325] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4518.652483] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4519.352299] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4530.232831] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4536.621363] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4539.037656] Lustre: DEBUG MARKER: ADD OST3 [ 4544.473647] loop: module loaded [ 4544.602638] loop0: detected capacity change from 0 to 204800000 [ 4544.899976] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4544.912318] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4544.931091] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4552.531920] Lustre: DEBUG MARKER: STOP OST3 [ 4552.695814] loop0: detected capacity change from 0 to 204800000 [ 4560.212793] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4560.399885] Lustre: lustre-OST0003: new disk, initializing [ 4560.402755] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4560.414199] Lustre: 196152:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4561.888948] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4562.057401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4562.063155] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4562.117301] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4563.760922] Lustre: Failing over lustre-OST0003 [ 4563.936541] LustreError: 196251:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4564.164773] Lustre: server umount lustre-OST0003 complete [ 4567.023486] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4567.042138] LustreError: 193319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4567.065071] LustreError: 193319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4568.265401] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4577.425871] Lustre: DEBUG MARKER: START OST3 again [ 4584.657289] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4584.995495] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4585.010020] Lustre: Skipped 1 previous similar message [ 4585.042111] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4586.537468] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4586.615396] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4586.617885] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4594.016764] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4616.781494] Lustre: Failing over lustre-OST0003 [ 4616.865039] Lustre: server umount lustre-OST0003 complete [ 4617.697447] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4617.701362] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4617.701681] LustreError: Skipped 1 previous similar message [ 4617.702782] LustreError: 195003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4617.702788] LustreError: 195003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4617.755374] Lustre: Skipped 2 previous similar messages [ 4636.140399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4636.149729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4636.172741] Lustre: Skipped 3 previous similar messages [ 4636.194260] Lustre: Skipped 3 previous similar messages [ 4642.798498] Lustre: server umount lustre-MDT0000 complete [ 4647.999114] LustreError: 191848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781013939 with bad export cookie 7329159270329411906 [ 4648.013143] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4655.010336] Lustre: server umount lustre-MDT0001 complete [ 4659.588655] Lustre: server umount lustre-OST0000 complete [ 4664.025456] Lustre: server umount lustre-OST0001 complete [ 4671.643716] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 4682.403772] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 4732.524320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4732.850089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4732.888484] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4732.934265] Lustre: lustre-MDT0000: new disk, initializing [ 4732.994989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4733.013149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4738.281861] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4752.451242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4752.622101] Lustre: 201499:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4752.662153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4752.667792] Lustre: Skipped 1 previous similar message [ 4752.719669] Lustre: lustre-MDT0001: new disk, initializing [ 4752.821210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4752.835836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4756.960138] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4765.540994] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4771.867083] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4780.119362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4780.429411] Lustre: 203246:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4782.233530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4782.400768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4787.024645] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4795.454514] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4800.846643] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4801.083592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4807.356412] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4807.794549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4813.287036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4813.300489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.309265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4813.309274] Lustre: Skipped 8 previous similar messages [ 4813.327170] LustreError: Skipped 1 previous similar message [ 4813.358211] Lustre: Skipped 3 previous similar messages [ 4816.506500] Lustre: server umount lustre-OST0000 complete [ 4827.403232] Lustre: server umount lustre-MDT0000 complete [ 4828.661902] LustreError: 202373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4828.695291] LustreError: 202373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 4831.660402] LustreError: 201492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781014123 with bad export cookie 7329159270329414202 [ 4831.675050] LustreError: 201492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4831.692278] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4832.635784] Lustre: server umount lustre-MDT0001 complete [ 4836.636699] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:08:47 (1781014127) [ 4856.751580] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 4868.518179] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 4912.593815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4913.055064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4913.160687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4913.168572] Lustre: Skipped 1 previous similar message [ 4913.344683] Lustre: lustre-MDT0000: new disk, initializing [ 4913.348895] Lustre: Skipped 1 previous similar message [ 4913.590686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4913.606128] Lustre: Skipped 1 previous similar message [ 4918.960587] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4930.209092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4930.411244] Lustre: 207674:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4930.966414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4935.394399] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4943.893023] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4949.220025] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4958.132551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4958.377947] Lustre: 209418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4960.510055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4963.931518] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4970.989862] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4975.674852] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4975.953268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4980.773931] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4981.093277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4986.337455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4986.338116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.350192] LustreError: Skipped 1 previous similar message [ 4986.360829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4986.374136] Lustre: Skipped 3 previous similar messages [ 4986.406969] Lustre: Skipped 4 previous similar messages [ 4989.568572] Lustre: server umount lustre-OST0000 complete [ 5008.355529] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5009.176211] Lustre: server umount lustre-MDT0000 complete [ 5011.947322] LustreError: 207687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5011.980411] LustreError: 207687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5014.531424] LustreError: 207668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781014306 with bad export cookie 7329159270329415084 [ 5014.541561] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5014.550614] LustreError: 207668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5015.140344] Lustre: server umount lustre-MDT0001 complete [ 5026.957204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5027.967717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5027.983249] Lustre: Skipped 5 previous similar messages [ 5033.476506] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5043.894803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5049.630516] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5056.328363] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5060.398355] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5067.996398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5075.044253] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5081.572915] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5092.847811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5092.851880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5092.885056] Lustre: Skipped 3 previous similar messages [ 5106.144657] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5106.292781] Lustre: server umount lustre-OST0000 complete [ 5110.752431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5116.717784] Lustre: server umount lustre-MDT0000 complete [ 5120.309407] LustreError: 211470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781014412 with bad export cookie 7329159270329415959 [ 5120.324215] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5120.333574] LustreError: 211470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5120.741785] Lustre: server umount lustre-MDT0001 complete [ 5220.344613] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 5223.101551] Key type lgssc unregistered [ 5223.347517] LNet: 214795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5223.358478] LNetError: 214795:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5224.422352] LNet: Removed LNI 192.168.201.139@tcp [ 5225.164171] Key type .llcrypt unregistered [ 5225.170051] Key type ._llcrypt unregistered [ 5245.098376] Key type ._llcrypt registered [ 5245.099846] Key type .llcrypt registered [ 5245.175993] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 5256.571796] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 5257.220959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5257.356841] alg: No test for adler32 (adler32-zlib) [ 5258.329653] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 5258.507457] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 5260.143186] Key type lgssc registered [ 5261.116338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5288.904169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5288.952970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5290.161558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5290.189411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5290.259198] Lustre: lustre-MDT0000: new disk, initializing [ 5290.336681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5290.354490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5293.430437] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5300.975687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5301.058621] Lustre: 218095:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5301.122508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5301.125896] Lustre: Skipped 1 previous similar message [ 5301.179426] Lustre: lustre-MDT0001: new disk, initializing [ 5301.230199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5301.260632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5301.269286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5303.927316] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5309.886728] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5312.765514] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5317.334640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5317.503690] Lustre: lustre-OST0000: new disk, initializing [ 5317.505536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5317.508795] Lustre: 219834:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5317.560432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.007651] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5323.351221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5323.357924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5323.446170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5328.047913] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5331.483658] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5331.638235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5334.579424] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5334.731402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5339.111646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5339.112451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5339.119416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5339.133102] Lustre: Skipped 1 previous similar message [ 5344.226723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5344.231652] Lustre: Skipped 2 previous similar messages [ 5349.351874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5349.357558] Lustre: Skipped 1 previous similar message [ 5350.368234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5350.525121] Lustre: server umount lustre-OST0000 complete [ 5354.470655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5354.480892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5354.485861] Lustre: Skipped 1 previous similar message [ 5359.429936] Lustre: server umount lustre-MDT0000 complete [ 5362.047993] LustreError: 219833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781014654 with bad export cookie 6018854582609526586 [ 5362.051737] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5362.059578] LustreError: 219833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5362.365894] Lustre: server umount lustre-MDT0001 complete [ 5375.443201] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 5382.855317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5383.077040] LustreError: 222600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5383.083542] LustreError: 222600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5383.104440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5385.835437] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5388.268243] LustreError: 222601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5392.300696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5392.587676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5395.692683] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5397.984680] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5402.874232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5403.113693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5407.229843] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5413.621480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5413.764863] Lustre: lustre-OST0001: new disk, initializing [ 5413.767759] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5413.771964] Lustre: 224853:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5418.686992] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5420.608729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5420.617302] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5420.694238] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5428.337236] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5437.311675] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5446.081806] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:18:57 (1781014737) [ 5449.185209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5449.190251] LustreError: Skipped 1 previous similar message [ 5449.194178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5449.203975] Lustre: Skipped 1 previous similar message [ 5449.215876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5451.751297] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.767882] Lustre: Skipped 1 previous similar message [ 5453.730958] Lustre: server umount lustre-MDT0000 complete [ 5456.541171] LustreError: 222581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781014748 with bad export cookie 6018854582609527475 [ 5456.548472] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5456.557193] LustreError: 222581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5456.865665] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5456.866404] LustreError: 222597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5456.877664] Lustre: Skipped 1 previous similar message [ 5456.898525] LustreError: 222597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5457.083406] Lustre: server umount lustre-MDT0001 complete [ 5462.347843] Lustre: server umount lustre-OST0000 complete [ 5465.557870] Lustre: server umount lustre-OST0001 complete [ 5470.561947] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 5477.414663] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 5502.248056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.414866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5502.437569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5502.493712] Lustre: lustre-MDT0000: new disk, initializing [ 5502.563357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5502.569244] Lustre: Skipped 1 previous similar message [ 5502.583337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5505.693672] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5514.215335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5514.317178] Lustre: 228741:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5514.403756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5514.408270] Lustre: Skipped 1 previous similar message [ 5514.475374] Lustre: lustre-MDT0001: new disk, initializing [ 5514.566474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5514.572756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5517.864127] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5523.509744] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5526.012560] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5530.259936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5530.433631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5530.437043] Lustre: 230479:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5531.868314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5531.876731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5531.939071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5533.973837] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5538.808386] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5541.880658] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5542.074712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5544.960672] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5545.051787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5547.495120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5547.499485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5547.500260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5547.500264] Lustre: Skipped 4 previous similar messages [ 5547.504664] Lustre: Skipped 2 previous similar messages [ 5547.510955] LustreError: Skipped 1 previous similar message [ 5552.440805] Lustre: server umount lustre-OST0000 complete [ 5555.681446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5561.317806] Lustre: server umount lustre-MDT0000 complete [ 5562.856273] LustreError: 228751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5562.874050] LustreError: 228751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5563.708886] LustreError: 230490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781014855 with bad export cookie 6018854582609529610 [ 5563.713073] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5563.714201] LustreError: 230490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5564.020571] Lustre: server umount lustre-MDT0001 complete [ 5569.377820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.700161] LustreError: 232548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5569.755619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5569.758516] Lustre: Skipped 2 previous similar messages [ 5572.723286] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5578.410181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.699330] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5585.551324] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5588.082240] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5592.820305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5597.071523] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5600.720962] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5746.656240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5746.658604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5746.660956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5746.665555] Lustre: Skipped 3 previous similar messages [ 5746.668923] Lustre: Skipped 5 previous similar messages [ 5757.407127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5757.475878] Lustre: server umount lustre-OST0000 complete [ 5765.288352] Lustre: server umount lustre-MDT0000 complete [ 5767.023189] LustreError: 232529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781015059 with bad export cookie 6018854582609530492 [ 5767.024105] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5767.033867] LustreError: 232529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5767.239733] Lustre: server umount lustre-MDT0001 complete [ 5864.034176] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 5865.139042] Key type lgssc unregistered [ 5865.275387] LNet: 235797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5865.278525] LNetError: 235797:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5865.288554] LNet: Removed LNI 192.168.201.139@tcp [ 5865.623144] Key type .llcrypt unregistered [ 5865.624868] Key type ._llcrypt unregistered [ 5875.019095] Key type ._llcrypt registered [ 5875.020374] Key type .llcrypt registered [ 5875.064195] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 5882.257733] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 5882.644818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5882.689383] alg: No test for adler32 (adler32-zlib) [ 5883.572831] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 5883.678290] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 5885.287233] Key type lgssc registered [ 5885.835652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5919.760542] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 5933.665041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5933.678719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5935.229283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5935.321719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5935.445838] Lustre: lustre-MDT0000: new disk, initializing [ 5935.541793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5935.581104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5941.964632] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5953.247897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5953.373761] Lustre: 239792:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5953.417827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5953.424903] Lustre: Skipped 1 previous similar message [ 5953.472302] Lustre: lustre-MDT0001: new disk, initializing [ 5953.559313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5953.631083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5953.648706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5957.989528] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5962.288482] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5967.437924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5967.665834] Lustre: lustre-OST0000: new disk, initializing [ 5967.669261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5967.676938] Lustre: 241391:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5967.768535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5971.294430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5971.304769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5971.377856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5973.117878] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5980.911542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5981.045105] Lustre: lustre-OST0001: new disk, initializing [ 5981.048353] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5981.052687] Lustre: 242243:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5981.140717] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5985.825678] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5990.950872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5990.954913] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5991.047104] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5994.576632] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5999.458792] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6006.757672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.759416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6006.773101] Lustre: Skipped 2 previous similar messages [ 6008.474847] Lustre: server umount lustre-MDT0000 complete [ 6010.881402] LustreError: 241390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781015302 with bad export cookie 145163077985806718 [ 6010.886122] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6010.897405] LustreError: 241390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6011.221832] Lustre: server umount lustre-MDT0001 complete [ 6023.294081] Lustre: server umount lustre-OST0000 complete [ 6035.975677] Lustre: server umount lustre-OST0001 complete [ 6045.715359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6052.558487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6057.956984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6064.302201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6076.731667] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 6082.941185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6082.952315] Lustre: lustre-MDT0000: reset Object Index mappings [ 6083.284761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6085.952190] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6088.676648] LustreError: 247218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6088.695208] LustreError: 247218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6091.610427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6091.637707] Lustre: lustre-MDT0001: reset Object Index mappings [ 6091.984445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6094.881771] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6096.710730] Lustre: 248324:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6101.577622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6106.063910] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6109.036865] LustreError: 248676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6109.044756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6111.854586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6112.046485] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6112.049666] Lustre: Skipped 1 previous similar message [ 6116.157049] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6116.726759] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6122.635927] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6125.220197] Lustre: 250158:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6131.681536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6131.685253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6131.698868] Lustre: Skipped 3 previous similar messages [ 6131.707115] Lustre: Skipped 5 previous similar messages [ 6134.772228] Lustre: server umount lustre-MDT0000 complete [ 6136.803119] LustreError: 247212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6136.825206] LustreError: 247212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6137.752616] LustreError: 248701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781015429 with bad export cookie 145163077985813508 [ 6137.761879] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6137.767036] LustreError: 248701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6137.974108] Lustre: server umount lustre-MDT0001 complete [ 6150.181866] Lustre: server umount lustre-OST0000 complete [ 6162.170514] Lustre: server umount lustre-OST0001 complete [ 6173.378090] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:31:04 (1781015464) [ 6182.784571] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 6188.821529] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 6207.338495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6207.475189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6207.511321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6207.563749] Lustre: lustre-MDT0000: new disk, initializing [ 6207.623716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6207.642318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6209.729869] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6215.448309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6215.514753] Lustre: 254373:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6215.554725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6215.556716] Lustre: Skipped 1 previous similar message [ 6215.597534] Lustre: lustre-MDT0001: new disk, initializing [ 6215.665953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6215.671905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6217.476783] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6221.508759] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6223.405830] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6226.975986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6227.101970] Lustre: 256115:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6228.305040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6228.390531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6229.751168] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6233.891765] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6235.841233] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6235.954437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6237.922568] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6238.020425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6243.814624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6243.816353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6243.819346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6243.819354] Lustre: Skipped 1 previous similar message [ 6243.822225] LustreError: Skipped 1 previous similar message [ 6243.843097] Lustre: Skipped 1 previous similar message [ 6245.025181] Lustre: server umount lustre-OST0000 complete [ 6248.928669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6248.929241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6248.933490] Lustre: Skipped 1 previous similar message [ 6248.935719] Lustre: Skipped 1 previous similar message [ 6252.814905] Lustre: server umount lustre-MDT0000 complete [ 6254.048246] LustreError: 254385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6254.056082] LustreError: 254385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6255.025617] LustreError: 254367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781015547 with bad export cookie 145163077985815594 [ 6255.029264] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6255.036381] LustreError: 254367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6255.222560] Lustre: server umount lustre-MDT0001 complete [ 6267.260102] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 6273.513100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6273.716866] LustreError: 258883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6273.744755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6273.750201] Lustre: Skipped 2 previous similar messages [ 6275.784828] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6280.141388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6282.274628] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6283.558724] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6286.459702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6289.988553] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6293.986113] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6294.079123] Lustre: lustre-OST0001: new disk, initializing [ 6294.080804] Lustre: Skipped 1 previous similar message [ 6294.083201] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6294.086263] Lustre: Skipped 1 previous similar message [ 6294.089307] Lustre: 261136:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6295.259333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6295.263290] Lustre: Skipped 1 previous similar message [ 6295.265384] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6295.314746] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6296.631153] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6301.590197] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6303.138365] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6305.384258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6305.396838] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6308.833270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6308.835956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6308.839113] Lustre: Skipped 3 previous similar messages [ 6308.843053] Lustre: Skipped 4 previous similar messages [ 6314.216694] Lustre: server umount lustre-MDT0000 complete [ 6315.785605] LustreError: 258864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781015607 with bad export cookie 145163077985816483 [ 6315.790258] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6315.790803] LustreError: 258864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6315.932358] Lustre: server umount lustre-MDT0001 complete [ 6327.330689] Lustre: server umount lustre-OST0000 complete [ 6339.991700] Lustre: server umount lustre-OST0001 complete [ 6348.936475] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 6354.611475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6354.842986] LustreError: 264154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6354.852638] LustreError: 264154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6357.135198] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6361.839473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6364.141791] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6365.511374] Lustre: 265260:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6368.870108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6371.112594] LustreError: 265612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6371.124261] LustreError: 265612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6372.144465] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6376.517947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6379.674496] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6379.777711] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6379.790592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6384.322030] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6386.128136] Lustre: 267099:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6388.940573] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6388.977360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6393.975143] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 10:34:45 (1781015685) [ 6408.160037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6408.163692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6408.171249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6408.173644] Lustre: Skipped 4 previous similar messages [ 6412.302170] Lustre: server umount lustre-MDT0000 complete [ 6414.143747] LustreError: 264136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781015706 with bad export cookie 145163077985822804 [ 6414.147269] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6414.150120] LustreError: 264136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6415.841061] LustreError: 264151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6415.850565] LustreError: 264151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6420.366730] Lustre: server umount lustre-MDT0001 complete [ 6428.374382] Lustre: server umount lustre-OST0000 complete [ 6436.403648] Lustre: server umount lustre-OST0001 complete [ 6439.306094] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 6443.748059] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 6464.852797] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 6470.124100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6470.272861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6470.312301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6470.361962] Lustre: lustre-MDT0000: new disk, initializing [ 6470.396911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6470.399322] Lustre: Skipped 7 previous similar messages [ 6470.415279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6472.953302] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6478.375600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6478.421115] Lustre: 271283:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6478.546024] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6480.243642] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6485.907409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6486.046871] Lustre: 272882:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6487.764898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6488.785133] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6494.417259] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6494.491158] Lustre: 273734:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6496.147855] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6497.192764] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6502.366978] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6503.856442] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6503.858933] Lustre: Skipped 1 previous similar message [ 6508.797438] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 10:36:40 (1781015800) [ 6511.584635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6511.587881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6511.591636] Lustre: Skipped 8 previous similar messages [ 6511.594009] Lustre: Skipped 10 previous similar messages [ 6515.834231] Lustre: server umount lustre-MDT0000 complete [ 6516.706352] LustreError: 271291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6516.712036] LustreError: 271291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6517.719994] LustreError: 272072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781015809 with bad export cookie 145163077985830693 [ 6517.720564] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6517.728328] LustreError: 272072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6517.889329] Lustre: server umount lustre-MDT0001 complete [ 6530.069512] Lustre: server umount lustre-OST0000 complete [ 6541.989284] Lustre: server umount lustre-OST0001 complete [ 6545.530696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6547.293487] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6550.592784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.387093] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6554.510346] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6555.860988] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6558.505895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6560.795401] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6563.037807] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6580.255137] loop1: detected capacity change from 0 to 204800000 [ 6580.373608] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6580.379797] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6580.387423] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6582.419329] loop1: detected capacity change from 0 to 204800000 [ 6584.700307] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6584.737977] LustreError: 275951:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs7270: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6584.743979] LustreError: 275951:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs7270 [ 6584.746769] LustreError: 275951:0:(mgs_handler.c:517:mgs_target_reg()) fs7270-MDT0001: Error checking target index: rc = -13 [ 6584.750940] LustreError: fs7270-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6584.754599] LustreError: 279356:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 6584.757817] LustreError: 279356:0:(tgt_mount.c:2084:server_put_super()) no obd fs7270-MDT0001 [ 6584.760379] LustreError: 279356:0:(tgt_mount.c:116:server_deregister_mount()) fs7270-MDT0001 not registered [ 6584.773686] Lustre: server umount fs7270-MDT0001 complete [ 6584.776342] LustreError: 279356:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6588.092861] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6588.129174] Lustre: Setting parameter fs7270-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs7270-MDT0001 [ 6588.151211] Lustre: srv-fs7270-MDT0001: No data found on store. Initialize space. [ 6588.153394] Lustre: Skipped 4 previous similar messages [ 6588.197552] Lustre: fs7270-MDT0001: new disk, initializing [ 6588.199595] Lustre: Skipped 3 previous similar messages [ 6588.234520] Lustre: srv-fs7270-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6588.237992] Lustre: 279823:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7270-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6588.241429] LustreError: 279823:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7270-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6589.927898] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6593.122879] Lustre: server umount fs7270-MDT0001 complete [ 6598.257908] loop1: detected capacity change from 0 to 204800000 [ 6600.487248] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6600.624625] Lustre: 281371:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7270-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6600.628757] LustreError: 281371:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7270-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6602.206475] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6604.326228] Lustre: server umount fs7270-MDT0001 complete [ 6609.501648] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) [ 6609.503959] Lustre: Skipped 1 previous similar message [ 6611.937151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.941214] Lustre: Skipped 2 previous similar messages [ 6614.759743] Lustre: server umount lustre-MDT0000 complete [ 6618.143613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6618.214285] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6619.939350] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6622.958918] loop1: detected capacity change from 0 to 204800000 [ 6623.720693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6625.121204] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6625.255745] Lustre: 283778:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7270-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6625.259719] LustreError: 283778:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7270-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6626.698371] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6628.464459] Lustre: server umount fs7270-MDT0001 complete [ 6637.536541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6641.624196] Lustre: server umount lustre-OST0000 complete [ 6644.193817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6644.196993] LustreError: Skipped 1 previous similar message [ 6649.195559] Lustre: server umount lustre-MDT0000 complete [ 6649.312237] LustreError: 278121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6649.317288] LustreError: 278121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 47 previous similar messages [ 6650.659268] LustreError: 275943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781015942 with bad export cookie 145163077985833367 [ 6650.663530] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6650.664204] LustreError: 275943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6650.806342] Lustre: server umount lustre-MDT0001 complete [ 6654.848047] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 6656.021883] Key type lgssc unregistered [ 6656.180395] LNet: 286082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6656.184356] LNetError: 286082:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6656.192355] LNet: Removed LNI 192.168.201.139@tcp [ 6656.544153] Key type .llcrypt unregistered [ 6656.545725] Key type ._llcrypt unregistered [ 6665.286592] Key type ._llcrypt registered [ 6665.287895] Key type .llcrypt registered [ 6665.334784] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 6665.832341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6665.840141] alg: No test for adler32 (adler32-zlib) [ 6666.741698] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 6666.851487] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 6668.447152] Key type lgssc registered [ 6668.883643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6672.001528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6672.011767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.215053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6674.820704] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6678.174084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6678.404781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6679.900058] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6681.917304] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6683.256881] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6685.672869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6685.812805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6687.860158] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6688.866077] LustreError: 288913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6688.872427] LustreError: 288913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6689.889270] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6694.369321] LustreError: 288913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6698.471138] LustreError: 288915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6698.480191] LustreError: 288915:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6700.649736] LustreError: 288915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6705.750532] LustreError: 288913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6705.758153] LustreError: 288913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6707.596386] loop1: detected capacity change from 0 to 204800000 [ 6707.665851] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6707.669415] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6707.673918] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6709.580786] loop1: detected capacity change from 0 to 204800000 [ 6711.680268] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6711.737610] Lustre: fs19913-OST0000: new disk, initializing [ 6711.739596] Lustre: srv-fs19913-OST0000: No data found on store. Initialize space. [ 6711.742741] Lustre: 290726:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6711.769319] Lustre: fs19913-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6713.827618] LustreError: 289379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6713.829841] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6713.832494] LustreError: 289379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6717.891672] Lustre: server umount fs19913-OST0000 complete [ 6722.583417] loop1: detected capacity change from 0 to 204800000 [ 6724.725111] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6724.810438] Lustre: fs19913-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.883873] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6730.172044] loop2: detected capacity change from 0 to 204800000 [ 6730.311094] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6730.314555] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6730.319303] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6731.349922] LustreError: 288915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6731.357028] LustreError: 288915:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6732.215023] loop2: detected capacity change from 0 to 204800000 [ 6734.460681] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6734.493656] LustreError: fs19913-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6734.497750] LustreError: fs19913-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6734.501601] LustreError: 293726:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 6734.504791] LustreError: 293726:0:(tgt_mount.c:2084:server_put_super()) no obd fs19913-OST0001 [ 6734.507353] LustreError: 293726:0:(tgt_mount.c:116:server_deregister_mount()) fs19913-OST0001 not registered [ 6734.523178] Lustre: server umount fs19913-OST0001 complete [ 6734.525053] LustreError: 293726:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6738.655735] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6738.706050] Lustre: fs19913-OST0001: new disk, initializing [ 6738.708494] Lustre: srv-fs19913-OST0001: No data found on store. Initialize space. [ 6738.712243] Lustre: 294276:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6738.737554] Lustre: fs19913-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6740.998413] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6744.806670] Lustre: server umount fs19913-OST0000 complete [ 6748.212913] Lustre: server umount fs19913-OST0001 complete [ 6752.895548] loop1: detected capacity change from 0 to 204800000 [ 6754.956583] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6755.044844] Lustre: fs19913-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6757.203542] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6759.731504] loop2: detected capacity change from 0 to 204800000 [ 6761.808147] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6763.884467] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6765.029529] LustreError: 288913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6765.034436] LustreError: 288913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 6767.196336] Lustre: server umount fs19913-OST0000 complete [ 6770.635635] Lustre: server umount fs19913-OST0001 complete [ 6775.265154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6775.265312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6775.266637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6775.275358] Lustre: Skipped 1 previous similar message [ 6780.385954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6780.387444] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 6780.388275] Lustre: Skipped 2 previous similar messages [ 6781.140204] Lustre: server umount lustre-OST0000 complete [ 6785.506683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6785.508473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6785.514592] Lustre: Skipped 1 previous similar message [ 6785.519939] Lustre: Skipped 1 previous similar message [ 6788.643614] Lustre: server umount lustre-MDT0000 complete [ 6789.997274] LustreError: 287295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781016081 with bad export cookie 14936645697696210418 [ 6790.001599] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6790.005070] LustreError: 287295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6790.148927] Lustre: server umount lustre-MDT0001 complete [ 6794.338660] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 6795.470412] Key type lgssc unregistered [ 6795.604084] LNet: 300118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6795.608082] LNetError: 300118:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6795.617316] LNet: Removed LNI 192.168.201.139@tcp [ 6795.947187] Key type .llcrypt unregistered [ 6795.948379] Key type ._llcrypt unregistered [ 6807.222990] Key type ._llcrypt registered [ 6807.223928] Key type .llcrypt registered [ 6807.265405] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 6813.751139] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 6814.195839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6814.207962] alg: No test for adler32 (adler32-zlib) [ 6815.076922] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 6815.162028] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 6816.751191] Key type lgssc registered [ 6817.158475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6830.083180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6830.089182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6831.191305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6831.208223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6831.249235] Lustre: lustre-MDT0000: new disk, initializing [ 6831.272090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6831.279595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6832.761786] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6837.360690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6837.415388] Lustre: 303914:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6837.446279] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6837.448473] Lustre: Skipped 1 previous similar message [ 6837.486537] Lustre: lustre-MDT0001: new disk, initializing [ 6837.521653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6837.536657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6837.541532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6839.045125] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6842.238063] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6843.485550] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6846.265470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6846.375208] Lustre: lustre-OST0000: new disk, initializing [ 6846.376651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6846.378882] Lustre: 305656:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6846.417341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6848.516079] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6851.942888] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6853.632505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6853.640235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6853.688235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6858.744785] Lustre: Mounted lustre-client [ 6859.158623] Lustre: Unmounted lustre-client [ 6863.842458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6863.842611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.846488] LustreError: Skipped 1 previous similar message [ 6863.847070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6863.850544] Lustre: Skipped 1 previous similar message [ 6868.564771] Lustre: server umount lustre-OST0000 complete [ 6873.567853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6873.570519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6873.575237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6873.577891] Lustre: Skipped 1 previous similar message [ 6876.349092] Lustre: server umount lustre-MDT0000 complete [ 6877.823717] LustreError: 303907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781016169 with bad export cookie 12386037454228517372 [ 6877.825575] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6877.828157] LustreError: 303907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6877.973865] Lustre: server umount lustre-MDT0001 complete [ 6882.323420] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 6883.604311] Key type lgssc unregistered [ 6907.359187] LNet: 1 local NIs in recovery (showing 1): 192.168.201.139@tcp [ 6908.319429] LNet: 307684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6908.321904] LNetError: 307684:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6985.190853] LNet: Removed LNI 192.168.201.139@tcp [ 6986.656389] Key type .llcrypt unregistered [ 6986.659096] Key type ._llcrypt unregistered [ 7009.328607] Key type ._llcrypt registered [ 7009.331553] Key type .llcrypt registered [ 7009.492623] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 7021.389760] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7022.313604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7022.350081] alg: No test for adler32 (adler32-zlib) [ 7023.455128] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 7023.714964] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 7025.343163] Key type lgssc registered [ 7026.046145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7047.661318] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 10:45:38 (1781016338) [ 7051.855402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7051.866141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7053.048500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7053.090297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7053.170518] Lustre: lustre-MDT0000: new disk, initializing [ 7053.232170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7053.257590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7055.895877] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7062.943082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7063.020444] Lustre: 310854:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7063.057888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7063.060870] Lustre: Skipped 1 previous similar message [ 7063.103148] Lustre: lustre-MDT0001: new disk, initializing [ 7063.177935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7063.202642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7063.209973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7065.962652] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7071.787668] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7074.413483] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7078.977556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7079.198187] Lustre: lustre-OST0000: new disk, initializing [ 7079.201896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7079.206408] Lustre: 312592:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7079.299734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7083.266449] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7088.713651] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7089.175161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7089.184860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7089.236614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7104.485962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7104.486695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7104.491060] LustreError: Skipped 1 previous similar message [ 7104.491722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7104.503183] Lustre: Skipped 1 previous similar message [ 7109.608949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7109.615662] Lustre: Skipped 2 previous similar messages [ 7109.810430] Lustre: server umount lustre-OST0000 complete [ 7114.721367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7114.727736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7114.735862] Lustre: Skipped 1 previous similar message [ 7118.623846] Lustre: server umount lustre-MDT0000 complete [ 7121.133067] LustreError: 310844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781016413 with bad export cookie 8518505263899574596 [ 7121.134110] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7121.139546] LustreError: 310844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7121.347603] Lustre: server umount lustre-MDT0001 complete [ 7128.081859] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7129.868734] Key type lgssc unregistered [ 7130.068557] LNet: 314371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7130.077065] LNetError: 314371:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7130.085764] LNet: Removed LNI 192.168.201.139@tcp [ 7130.607842] Key type .llcrypt unregistered [ 7130.616063] Key type ._llcrypt unregistered [ 7144.957199] Key type ._llcrypt registered [ 7144.959474] Key type .llcrypt registered [ 7145.024764] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7145.721753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7145.734153] alg: No test for adler32 (adler32-zlib) [ 7146.695366] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 7146.885231] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 7148.535136] Key type lgssc registered [ 7149.377213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7154.785452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7154.802033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.172069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7158.898247] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7164.987151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7165.216451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7168.275574] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7172.359544] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7175.444617] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7180.614391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7180.789487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7184.795959] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7188.950651] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7198.232347] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7199.129490] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7205.991855] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) [ 7206.370266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7206.379649] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7206.391894] Lustre: Skipped 2 previous similar messages [ 7211.096827] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) [ 7211.101160] Lustre: Skipped 3 previous similar messages [ 7216.095580] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7216.199673] Lustre: server umount lustre-MDT0000 complete [ 7216.216480] LustreError: 316320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7216.234086] LustreError: 316320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7219.232639] LustreError: 315589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781016511 with bad export cookie 5085700314276186729 [ 7219.247525] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7221.335465] LustreError: 315603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7221.345114] LustreError: 315603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7221.350114] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.39@tcp (stopping) [ 7221.353754] Lustre: Skipped 3 previous similar messages [ 7221.733948] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7224.738308] LustreError: 315605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.738447] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7224.748782] LustreError: 315605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7224.753735] Lustre: Skipped 1 previous similar message [ 7225.563677] Lustre: server umount lustre-MDT0001 complete [ 7228.482728] Lustre: server umount lustre-OST0000 complete [ 7234.271876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7236.632370] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7238.306754] Lustre: 319332:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.139@tcp [ 7239.448700] Lustre: server umount MGS complete [ 7244.115611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7244.385191] LustreError: 319902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7244.442076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7246.977518] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7249.888266] LustreError: 319903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7251.872076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7252.056214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7254.502778] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7259.372569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7262.535034] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7265.617593] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7266.563840] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7268.490392] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7273.617609] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 7277.025475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7277.030034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7277.042814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7277.398096] Lustre: server umount lustre-OST0000 complete [ 7280.098542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7280.107772] Lustre: Skipped 1 previous similar message [ 7285.216369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7285.219854] Lustre: Skipped 3 previous similar messages [ 7289.369452] Lustre: server umount lustre-MDT0000 complete [ 7290.337722] LustreError: 319898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7290.346613] LustreError: 319898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7291.641894] LustreError: 319883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781016583 with bad export cookie 5085700314276187569 [ 7291.647150] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7291.649783] LustreError: 319883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7291.862407] Lustre: server umount lustre-MDT0001 complete [ 7297.288380] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7298.878647] Key type lgssc unregistered [ 7299.087448] LNet: 323075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7299.091747] LNetError: 323075:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7299.105194] LNet: Removed LNI 192.168.201.139@tcp [ 7299.537813] Key type .llcrypt unregistered [ 7299.539585] Key type ._llcrypt unregistered [ 7309.297105] Key type ._llcrypt registered [ 7309.298871] Key type .llcrypt registered [ 7309.356991] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 7319.005025] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7319.659540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7319.679133] alg: No test for adler32 (adler32-zlib) [ 7320.610086] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 7320.718480] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 7322.312980] Key type lgssc registered [ 7322.966360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7344.098819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7344.111164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7345.249418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7345.269657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7345.318575] Lustre: lustre-MDT0000: new disk, initializing [ 7345.369640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7345.398295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7348.023404] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7354.318845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7354.432391] Lustre: 326180:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7354.466467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7354.469524] Lustre: Skipped 1 previous similar message [ 7354.518191] Lustre: lustre-MDT0001: new disk, initializing [ 7354.604518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7354.630536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7354.634979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7356.896321] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7361.529052] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7363.570417] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7368.166771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7368.379345] Lustre: lustre-OST0000: new disk, initializing [ 7368.382450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7368.387604] Lustre: 327918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7368.450095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7372.004373] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7376.944620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7376.951939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7377.022282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7377.202849] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7379.487763] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7379.596576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7382.028074] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7382.137976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7387.104394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7387.107859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7387.112681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7387.618781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7387.619115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7387.624403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7389.303662] Lustre: server umount lustre-OST0000 complete [ 7392.740131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7392.750980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7392.755747] Lustre: Skipped 1 previous similar message [ 7397.733950] Lustre: server umount lustre-MDT0000 complete [ 7399.922791] LustreError: 327917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781016691 with bad export cookie 17963872346088379216 [ 7399.928103] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7399.931133] LustreError: 327917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7400.137569] Lustre: server umount lustre-MDT0001 complete [ 7405.651535] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 10:51:36 (1781016696) [ 7415.470712] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 7420.875815] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7443.397704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7443.566506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7443.589708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7443.635772] Lustre: lustre-MDT0000: new disk, initializing [ 7443.690439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7443.704173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7446.071722] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7452.349629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7452.418701] Lustre: 332489:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7452.462759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7452.464767] Lustre: Skipped 1 previous similar message [ 7452.520956] Lustre: lustre-MDT0001: new disk, initializing [ 7452.584904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7452.611796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7452.618490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7454.863993] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7459.037630] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7460.692971] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7464.050131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7464.184071] Lustre: lustre-OST0000: new disk, initializing [ 7464.185902] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7464.190103] Lustre: 334231:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7464.226711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7466.985179] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7469.084074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7469.086607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7469.132840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7471.098486] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7489.504587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7489.504802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7489.508260] LustreError: Skipped 1 previous similar message [ 7489.514865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7489.515940] Lustre: Skipped 2 previous similar messages [ 7491.944596] Lustre: server umount lustre-OST0000 complete [ 7494.624599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7494.626640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7494.635681] Lustre: Skipped 1 previous similar message [ 7494.638758] Lustre: Skipped 1 previous similar message [ 7499.937808] Lustre: server umount lustre-MDT0000 complete [ 7501.906151] LustreError: 333265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781016793 with bad export cookie 17963872346088380098 [ 7501.909345] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7501.910922] LustreError: 333265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7502.112520] Lustre: server umount lustre-MDT0001 complete [ 7507.044338] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7508.439558] Key type lgssc unregistered [ 7508.598499] LNet: 336250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7508.602419] LNetError: 336250:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7508.615782] LNet: Removed LNI 192.168.201.139@tcp [ 7509.087559] Key type .llcrypt unregistered [ 7509.089169] Key type ._llcrypt unregistered [ 7551.203738] Key type ._llcrypt registered [ 7551.205117] Key type .llcrypt registered [ 7551.260201] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7551.387379] Key type .llcrypt unregistered [ 7551.388462] Key type ._llcrypt unregistered [ 7584.116540] Key type ._llcrypt registered [ 7584.118350] Key type .llcrypt registered [ 7584.173364] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7584.317280] Key type .llcrypt unregistered [ 7584.320076] Key type ._llcrypt unregistered [ 7596.724934] Key type ._llcrypt registered [ 7596.726148] Key type .llcrypt registered [ 7596.779367] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7596.898352] Key type .llcrypt unregistered [ 7596.899963] Key type ._llcrypt unregistered [ 7609.553593] Key type ._llcrypt registered [ 7609.555197] Key type .llcrypt registered [ 7609.596137] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7609.751276] Key type .llcrypt unregistered [ 7609.753380] Key type ._llcrypt unregistered [ 7643.636387] Key type ._llcrypt registered [ 7643.637507] Key type .llcrypt registered [ 7643.678412] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7643.801375] Key type .llcrypt unregistered [ 7643.803834] Key type ._llcrypt unregistered [ 7657.210437] Key type ._llcrypt registered [ 7657.211620] Key type .llcrypt registered [ 7657.260078] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7657.393515] Key type .llcrypt unregistered [ 7657.395674] Key type ._llcrypt unregistered [ 7670.679257] Key type ._llcrypt registered [ 7670.680443] Key type .llcrypt registered [ 7670.723413] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7670.862423] Key type .llcrypt unregistered [ 7670.864343] Key type ._llcrypt unregistered [ 7683.801033] Key type ._llcrypt registered [ 7683.802310] Key type .llcrypt registered [ 7683.848750] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7683.990133] Key type .llcrypt unregistered [ 7683.991515] Key type ._llcrypt unregistered [ 7707.160361] Key type ._llcrypt registered [ 7707.164756] Key type .llcrypt registered [ 7707.321631] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7707.656601] Key type .llcrypt unregistered [ 7707.658683] Key type ._llcrypt unregistered [ 7737.215499] Key type ._llcrypt registered [ 7737.217765] Key type .llcrypt registered [ 7737.313890] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7737.534519] Key type .llcrypt unregistered [ 7737.536652] Key type ._llcrypt unregistered [ 7790.860318] Key type ._llcrypt registered [ 7790.862952] Key type .llcrypt registered [ 7790.933720] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7791.099319] Key type .llcrypt unregistered [ 7791.101318] Key type ._llcrypt unregistered [ 7811.159322] Key type ._llcrypt registered [ 7811.162238] Key type .llcrypt registered [ 7811.243527] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7811.413562] Key type .llcrypt unregistered [ 7811.415788] Key type ._llcrypt unregistered [ 7831.248318] Key type ._llcrypt registered [ 7831.250208] Key type .llcrypt registered [ 7831.323895] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7831.497384] Key type .llcrypt unregistered [ 7831.499706] Key type ._llcrypt unregistered [ 7868.366860] Key type ._llcrypt registered [ 7868.369191] Key type .llcrypt registered [ 7868.458046] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 7878.638696] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7879.294405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7879.372374] alg: No test for adler32 (adler32-zlib) [ 7880.325489] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 7880.495989] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 7882.127980] Key type lgssc registered [ 7882.885697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7908.171396] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7908.204505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7909.386479] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 7909.416227] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 7909.471298] Lustre: tst250fs-MDT0000: new disk, initializing [ 7909.541908] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7909.556649] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7912.927657] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7921.141693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7921.244549] Lustre: 355926:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7921.286876] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 7921.289939] Lustre: Skipped 1 previous similar message [ 7921.342876] Lustre: tst250fs-MDT0001: new disk, initializing [ 7921.417994] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7921.450103] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7921.457361] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7924.863530] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7928.782924] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7934.022993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7934.259508] Lustre: tst250fs-OST0000: new disk, initializing [ 7934.261868] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 7934.266295] Lustre: 357521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7934.334737] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7938.310623] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7939.636384] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7939.644802] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7939.783915] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7946.722902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7946.856807] Lustre: tst250fs-OST0001: new disk, initializing [ 7946.860045] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 7946.864458] Lustre: 358374:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7946.934644] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7949.839201] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7949.844373] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7949.887464] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7951.236967] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7955.634618] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 7972.833400] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7972.840446] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7972.858311] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7975.397369] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7975.400407] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7975.411730] Lustre: Skipped 1 previous similar message [ 7975.416747] Lustre: Skipped 3 previous similar messages [ 7977.216707] Lustre: server umount tst250fs-MDT0000 complete [ 7979.936994] LustreError: 355916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781017271 with bad export cookie 9244739633139417518 [ 7979.939853] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7979.944648] LustreError: 355916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7980.162299] Lustre: server umount tst250fs-MDT0001 complete [ 7993.372584] Lustre: server umount tst250fs-OST0000 complete [ 8006.702911] Lustre: server umount tst250fs-OST0001 complete [ 8021.750412] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 8028.129182] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8054.194247] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 11:02:25 (1781017345) [ 8056.562562] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8082.947649] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 8100.230674] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8144.914995] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7905 sec ========== 11:03:54 (1781017434) [ 8148.244634] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:03:57 (1781017437) === [ 8152.956248] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:04:03 (1781017443) === [ 8197.575463] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 8200.801125] Key type lgssc unregistered [ 8201.197942] LNet: 367435:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8201.205875] LNetError: 367435:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8201.227917] LNet: Removed LNI 192.168.201.139@tcp [ 8202.367178] Key type .llcrypt unregistered [ 8202.373592] Key type ._llcrypt unregistered