[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496698411 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003293] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008776] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011009] pid_max: default: 32768 minimum: 301 [ 0.012133] LSM: Security Framework initializing [ 0.013043] Yama: becoming mindful. [ 0.015020] SELinux: Initializing. [ 0.016071] *** VALIDATE selinux *** [ 0.024455] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028451] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031119] *** VALIDATE tmpfs *** [ 0.032459] *** VALIDATE proc *** [ 0.034036] *** VALIDATE cgroup *** [ 0.035007] *** VALIDATE cgroup2 *** [ 0.036256] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043190] debug: unmapping init [mem 0xffffffffb7459000-0xffffffffb7460fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046776] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047020] ... version: 2 [ 0.048010] ... bit width: 48 [ 0.049008] ... generic registers: 4 [ 0.050010] ... value mask: 0000ffffffffffff [ 0.051011] ... max period: 00007fffffffffff [ 0.052012] ... fixed-purpose events: 3 [ 0.053007] ... event mask: 000000070000000f [ 0.054288] rcu: Hierarchical SRCU implementation. [ 0.056700] smp: Bringing up secondary CPUs ... [ 0.057691] x86: Booting SMP configuration: [ 0.058118] .... node #0, CPUs: #1 #2 #3 [ 0.065314] smp: Brought up 1 node, 4 CPUs [ 0.067032] smpboot: Max logical packages: 1 [ 0.068012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130905] node 0 deferred pages initialised in 60ms [ 0.151123] devtmpfs: initialized [ 0.153337] x86/mm: Memory block size: 128MB [ 0.161975] gcov: version magic: 0x41383552 [ 0.168925] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.176093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.179715] pinctrl core: initialized pinctrl subsystem [ 0.182357] [ 0.183007] ************************************************************* [ 0.186011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191029] ** ** [ 0.196081] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.200011] ** ** [ 0.203012] ** This means that this kernel is built to expose internal ** [ 0.207010] ** IOMMU data structures, which may compromise security on ** [ 0.211024] ** your system. ** [ 0.215018] ** ** [ 0.218016] ** If you see this message and you are not debugging the ** [ 0.221012] ** kernel, report this immediately to your vendor! ** [ 0.224015] ** ** [ 0.227020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.229009] ************************************************************* [ 0.234175] NET: Registered protocol family 16 [ 0.236674] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.240090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.245071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.251125] cpuidle: using governor menu [ 0.254179] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.257483] PCI: Using configuration type 1 for base access [ 0.260152] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.271201] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.279497] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.285505] cryptd: max_cpu_qlen set to 1000 [ 0.288266] ACPI: Added _OSI(Module Device) [ 0.289023] ACPI: Added _OSI(Processor Device) [ 0.290000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.290000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.297833] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.305677] ACPI: Interpreter enabled [ 0.307073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.309008] ACPI: Using IOAPIC for interrupt routing [ 0.310119] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.314499] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.326412] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.330041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.334023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.338140] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.347763] acpiphp: Slot [2] registered [ 0.350238] acpiphp: Slot [5] registered [ 0.359297] acpiphp: Slot [6] registered [ 0.361246] acpiphp: Slot [7] registered [ 0.362129] acpiphp: Slot [8] registered [ 0.365161] acpiphp: Slot [9] registered [ 0.373216] acpiphp: Slot [10] registered [ 0.377191] acpiphp: Slot [3] registered [ 0.380280] acpiphp: Slot [4] registered [ 0.382093] acpiphp: Slot [11] registered [ 0.383000] acpiphp: Slot [12] registered [ 0.383000] acpiphp: Slot [13] registered [ 0.385210] acpiphp: Slot [14] registered [ 0.386068] acpiphp: Slot [15] registered [ 0.387000] acpiphp: Slot [16] registered [ 0.387000] acpiphp: Slot [17] registered [ 0.389061] acpiphp: Slot [18] registered [ 0.390063] acpiphp: Slot [19] registered [ 0.392095] acpiphp: Slot [20] registered [ 0.393075] acpiphp: Slot [21] registered [ 0.395096] acpiphp: Slot [22] registered [ 0.396109] acpiphp: Slot [23] registered [ 0.398360] acpiphp: Slot [24] registered [ 0.400132] acpiphp: Slot [25] registered [ 0.401080] acpiphp: Slot [26] registered [ 0.402105] acpiphp: Slot [27] registered [ 0.403434] acpiphp: Slot [28] registered [ 0.409140] acpiphp: Slot [29] registered [ 0.412141] acpiphp: Slot [30] registered [ 0.417591] acpiphp: Slot [31] registered [ 0.420070] PCI host bridge to bus 0000:00 [ 0.421000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.422018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.425018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.428022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.430018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.432022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.434177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.436969] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.440189] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.451000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.454048] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.456015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.458012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.460013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.462596] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.466792] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.468031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.470821] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.475012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.489015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.494012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.500462] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.508014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.513015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.531015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.549567] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.558014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.564016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.579000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.673218] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.686015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.696021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.717016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.731000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.738014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.743015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.781019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.814013] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.823026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.833031] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.868017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.879355] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.888016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.895013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.922015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.932362] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.934272] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.935320] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.936206] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.938178] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.941051] iommu: Default domain type: Passthrough [ 0.944872] SCSI subsystem initialized [ 0.948173] ACPI: bus type USB registered [ 0.949082] usbcore: registered new interface driver usbfs [ 0.951120] usbcore: registered new interface driver hub [ 0.956106] usbcore: registered new device driver usb [ 0.957126] pps_core: LinuxPPS API ver. 1 registered [ 0.960014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.963060] PTP clock support registered [ 0.965364] EDAC MC: Ver: 3.0.0 [ 0.967106] PCI: Using ACPI for IRQ routing [ 0.970144] NetLabel: Initializing [ 0.973009] NetLabel: domain hash size = 128 [ 0.974008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.977085] NetLabel: unlabeled traffic allowed by default [ 0.980075] vgaarb: loaded [ 0.981357] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.983012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.989967] clocksource: Switched to clocksource kvm-clock [ 1.156468] VFS: Disk quotas dquot_6.6.0 [ 1.158045] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.161258] *** VALIDATE ramfs *** [ 1.162303] *** VALIDATE hugetlbfs *** [ 1.164239] pnp: PnP ACPI init [ 1.166857] pnp: PnP ACPI: found 6 devices [ 1.184835] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.187954] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.190924] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.193407] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.196067] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.198564] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.201530] NET: Registered protocol family 2 [ 1.204490] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.209701] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.213590] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.219180] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.222437] TCP: Hash tables configured (established 65536 bind 65536) [ 1.225684] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.229278] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.232332] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.235928] NET: Registered protocol family 1 [ 1.238932] RPC: Registered named UNIX socket transport module. [ 1.241600] RPC: Registered udp transport module. [ 1.243624] RPC: Registered tcp transport module. [ 1.245727] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.248469] NET: Registered protocol family 44 [ 1.250307] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.252513] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.254709] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.257449] PCI: CLS 0 bytes, default 64 [ 1.260158] Unpacking initramfs... [ 3.062686] debug: unmapping init [mem 0xffff92793cc54000-0xffff92793ffbffff] [ 3.069776] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.072656] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.077484] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.038653] Initialise system trusted keyrings [ 4.040233] Key type blacklist registered [ 4.045516] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.069080] zbud: loaded [ 4.078254] *** VALIDATE nfs *** [ 4.079422] *** VALIDATE nfs4 *** [ 4.090070] pstore: using deflate compression [ 4.100365] Platform Keyring initialized [ 4.466583] NET: Registered protocol family 38 [ 4.472377] Key type asymmetric registered [ 4.476559] Asymmetric key parser 'x509' registered [ 4.480671] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.487332] io scheduler mq-deadline registered [ 4.489075] io scheduler kyber registered [ 4.491355] io scheduler bfq registered [ 4.493635] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.497181] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.503569] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.531397] ACPI: Power Button [PWRF] [ 4.554492] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.581883] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.618165] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.637539] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.669841] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.724204] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.771608] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.781130] Non-volatile memory driver v1.3 [ 4.785904] Linux agpgart interface v0.103 [ 4.882842] virtio_blk virtio1: [vda] 135992 512-byte logical blocks (69.6 MB/66.4 MiB) [ 4.892103] vda: detected capacity change from 0 to 69627904 [ 4.928756] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.937411] vdb: detected capacity change from 0 to 1073741824 [ 4.989310] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.997711] vdc: detected capacity change from 0 to 2621440000 [ 5.030639] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.037752] vdd: detected capacity change from 0 to 2621440000 [ 5.076788] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.086017] vde: detected capacity change from 0 to 4294967296 [ 5.132870] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.141681] vdf: detected capacity change from 0 to 4294967296 [ 5.177628] libphy: Fixed MDIO Bus: probed [ 5.198303] usbcore: registered new interface driver usbserial_generic [ 5.200944] usbserial: USB Serial support registered for generic [ 5.210691] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.225291] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.230671] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.242230] mousedev: PS/2 mouse device common for all mice [ 5.256047] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.262575] rtc_cmos 00:05: RTC can wake from S4 [ 5.281139] rtc_cmos 00:05: registered as rtc0 [ 5.294753] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.297516] intel_pstate: CPU model not supported [ 5.336355] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.343969] hid: raw HID events driver (C) Jiri Kosina [ 5.359666] usbcore: registered new interface driver usbhid [ 5.362994] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.376089] usbhid: USB HID core driver [ 5.376309] drop_monitor: Initializing network drop monitor service [ 5.385622] Initializing XFRM netlink socket [ 5.413164] NET: Registered protocol family 10 [ 5.434814] Segment Routing with IPv6 [ 5.450377] NET: Registered protocol family 17 [ 5.458493] mpls_gso: MPLS GSO support [ 5.477761] RAS: Correctable Errors collector initialized. [ 5.484663] AVX version of gcm_enc/dec engaged. [ 5.489909] AES CTR mode by8 optimization enabled [ 5.870037] sched_clock: Marking stable (5870015232, 0)->(6879021704, -1009006472) [ 5.885414] registered taskstats version 1 [ 5.895625] Loading compiled-in X.509 certificates [ 5.904871] zswap: loaded using pool lzo/zbud [ 6.020884] Key type big_key registered [ 6.121650] Key type encrypted registered [ 6.124663] ima: No TPM chip found, activating TPM-bypass! [ 6.130164] ima: Allocated hash algorithm: sha1 [ 6.137690] ima: No architecture policies found [ 6.142221] evm: Initialising EVM extended attributes: [ 6.144223] evm: security.selinux [ 6.148838] evm: security.ima [ 6.152600] evm: security.capability [ 6.159205] evm: HMAC attrs: 0x1 [ 6.164772] rtc_cmos 00:05: setting system clock to 2026-06-04 00:33:32 UTC (1780533212) [ 6.181124] debug: unmapping init [mem 0xffffffffb8403000-0xffffffffb85fffff] [ 6.202076] debug: unmapping init [mem 0xffffffffb7182000-0xffffffffb7458fff] [ 6.213251] Write protecting the kernel read-only data: 28672k [ 6.222572] debug: unmapping init [mem 0xffffffffb5803000-0xffffffffb59fffff] [ 6.228649] debug: unmapping init [mem 0xffffffffb6114000-0xffffffffb61fffff] [ 6.376780] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.411890] systemd[1]: Detected virtualization kvm. [ 6.415785] systemd[1]: Detected architecture x86-64. [ 6.422253] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.540057] systemd[1]: No hostname configured. [ 6.542643] systemd[1]: Set hostname to . [ 6.547286] random: systemd: uninitialized urandom read (16 bytes read) [ 6.556791] systemd[1]: Initializing machine ID from random generator. [ 6.769090] random: ln: uninitialized urandom read (6 bytes read) [ 7.074069] random: systemd: uninitialized urandom read (16 bytes read) [ 7.091146] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 7.103386] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 7.110318] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.248835] device-mapper: uevent: version 1.0.3 [ 9.262223] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.134884] virtio_net virtio0 ens2: renamed from eth0 [ 12.429373] random: fast init done [ 12.904637] scsi host0: ata_piix [ 13.148617] scsi host1: ata_piix [ 13.156695] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.175259] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.679945] random: crng init done [ 17.686442] random: 7 urandom warning(s) missed due to ratelimiting [ 20.033196] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 22.509519] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.507538] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.874975] SELinux: Disabled at runtime. [ 27.052868] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.071368] systemd[1]: Detected virtualization kvm. [ 27.077890] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.700976] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.706250] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.718108] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.724602] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.727444] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.757443] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.811919] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ 29.486564] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 31.081663] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.314745] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 32.551907] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.933469] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.983767] EDAC sbridge: Ver: 1.1.2 [ 35.585485] Key type dns_resolver registered [ 36.063332] NFS: Registering the id_resolver key type [ 36.065349] Key type id_resolver registered [ 36.066869] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg303-server login: [ 50.246245] hrtimer: interrupt took 8236857 ns [ 78.291556] libcfs: loading out-of-tree module taints kernel. [ 78.313681] Key type ._llcrypt registered [ 78.315273] Key type .llcrypt registered [ 78.361590] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 87.076636] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 87.734083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 87.740404] alg: No test for adler32 (adler32-zlib) [ 88.757303] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 89.084698] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 90.703547] Key type lgssc registered [ 91.326502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 98.983599] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 118.177308] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 123.506678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 123.522863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 124.655933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 124.674504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 124.732966] Lustre: lustre-MDT0000: new disk, initializing [ 124.788177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 124.804282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 126.562729] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 133.148699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 133.212965] Lustre: 6481:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 133.231412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 133.235148] Lustre: Skipped 1 previous similar message [ 133.280760] Lustre: lustre-MDT0001: new disk, initializing [ 133.314630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 133.328884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 133.332169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 134.863254] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 137.497157] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 141.093789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 141.196790] Lustre: lustre-OST0000: new disk, initializing [ 141.199270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 141.203235] Lustre: 8377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 141.226319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 141.702344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 141.706040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 141.732869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 143.436337] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 150.244166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 150.318237] Lustre: lustre-OST0001: new disk, initializing [ 150.321758] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 150.325618] Lustre: 9443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 150.357230] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 153.307215] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 155.645224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 155.649867] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 155.674482] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400 [ 160.512469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 166.850869] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 169.391259] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing check_logdir /tmp/testlogs/ [ 171.160537] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing yml_node [ 172.925484] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 173.851281] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 174.762257] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 175.353538] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jun 3 20:36:22 EDT 2026 [ 181.850753] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 182.480953] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 194.528590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 194.528970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 194.535025] Lustre: Skipped 3 previous similar messages [ 194.539387] Lustre: Skipped 2 previous similar messages [ 199.647858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 199.651013] Lustre: Skipped 3 previous similar messages [ 200.674729] Lustre: server umount lustre-MDT0000 complete [ 202.334258] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533408 with bad export cookie 10609290483794176391 [ 202.336103] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 202.339501] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 204.767679] LustreError: 6487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 204.768464] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 204.769535] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 204.782226] LustreError: 6487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 204.785923] Lustre: Skipped 1 previous similar message [ 207.775873] LustreError: 7991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 207.775993] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 207.790135] LustreError: 7991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 207.798460] Lustre: Skipped 2 previous similar messages [ 208.544467] Lustre: server umount lustre-MDT0001 complete [ 210.270390] Lustre: server umount lustre-OST0000 complete [ 212.102248] Lustre: server umount lustre-OST0001 complete [ 215.105839] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 220.325248] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 238.181492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.329875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 238.351857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 238.410276] Lustre: lustre-MDT0000: new disk, initializing [ 238.468616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.484859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 240.523109] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 246.416930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 246.487908] Lustre: 14947:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 246.528740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 246.531896] Lustre: Skipped 1 previous similar message [ 246.598742] Lustre: lustre-MDT0001: new disk, initializing [ 246.665541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.701854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 246.709865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 248.705046] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 252.815332] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 254.571409] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 257.444891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 257.572263] Lustre: 16691:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.951154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 259.003264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 259.853401] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 263.267587] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 265.011963] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 265.111228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 266.677225] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 266.757735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 269.281050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 269.283933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.284736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 269.288587] LustreError: Skipped 1 previous similar message [ 269.306583] Lustre: Skipped 1 previous similar message [ 273.664272] Lustre: server umount lustre-OST0000 complete [ 277.473713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 277.478183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 277.487636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 277.495748] Lustre: Skipped 1 previous similar message [ 281.569765] Lustre: server umount lustre-MDT0000 complete [ 283.320514] LustreError: 15727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533489 with bad export cookie 10609290483794177847 [ 283.322736] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 283.325026] LustreError: 15727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 283.500958] Lustre: server umount lustre-MDT0001 complete [ 286.216679] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 20:38:13 (1780533493) [ 294.348311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 294.589623] LustreError: 19416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.629098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 294.632524] Lustre: Skipped 1 previous similar message [ 296.343662] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 300.001413] LustreError: 19417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.383266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 302.414974] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 304.942174] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 306.627106] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 307.631138] Lustre: Failing over lustre-MDT0000 [ 307.749854] Lustre: server umount lustre-MDT0000 complete [ 309.728552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.733818] LustreError: 19416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.738496] Lustre: Skipped 2 previous similar messages [ 320.993602] LustreError: 19416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.002299] LustreError: 19416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 322.073105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 322.144364] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 324.106218] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 326.368744] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 326.372585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 326.379193] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 328.278445] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 331.745844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.746846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.756436] Lustre: Skipped 1 previous similar message [ 331.757815] Lustre: Skipped 2 previous similar messages [ 331.761103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 336.812594] Lustre: server umount lustre-MDT0000 complete [ 338.564741] LustreError: 21735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533544 with bad export cookie 10609290483794179268 [ 338.569303] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.572198] LustreError: 21735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 338.728694] Lustre: server umount lustre-MDT0001 complete [ 342.999965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 343.264434] LustreError: 22413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.276893] LustreError: 22413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 343.316332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 343.320383] Lustre: Skipped 2 previous similar messages [ 345.215279] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 349.256332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 351.292431] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 353.793070] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 355.378107] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 356.278338] Lustre: Failing over lustre-MDT0000 [ 356.382167] Lustre: server umount lustre-MDT0000 complete [ 358.880700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.886921] Lustre: Skipped 1 previous similar message [ 370.393380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 370.465672] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 372.390657] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 374.692615] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 374.695810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 374.701337] Lustre: Skipped 1 previous similar message [ 374.709427] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 376.443042] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 379.872762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 379.878842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 379.881231] Lustre: Skipped 1 previous similar message [ 393.183317] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 393.280388] Lustre: server umount lustre-MDT0000 complete [ 395.076230] LustreError: 22395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533601 with bad export cookie 10609290483794180206 [ 395.077208] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 395.086239] LustreError: 22395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 395.220821] Lustre: server umount lustre-MDT0001 complete [ 399.487959] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 20:40:06 (1780533606) [ 407.055368] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 411.337985] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 428.044258] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 433.007905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.115954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 433.130272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 433.133609] Lustre: Skipped 1 previous similar message [ 433.171040] Lustre: lustre-MDT0000: new disk, initializing [ 433.173311] Lustre: Skipped 1 previous similar message [ 433.219417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 433.222664] Lustre: Skipped 2 previous similar messages [ 433.230702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 433.234674] Lustre: Skipped 1 previous similar message [ 435.157136] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 440.344938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.405786] Lustre: 28390:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 440.541752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 442.483403] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 446.257310] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 447.996086] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 451.195918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 451.338912] Lustre: 30134:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 453.412587] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 453.445752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 453.453459] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 454.025506] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 457.942539] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 461.152270] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 468.962089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 468.962352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.965247] LustreError: Skipped 1 previous similar message [ 468.971126] Lustre: Skipped 2 previous similar messages [ 468.972064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 468.977869] Lustre: Skipped 5 previous similar messages [ 474.225583] Lustre: server umount lustre-OST0000 complete [ 482.041508] Lustre: server umount lustre-MDT0000 complete [ 483.701713] LustreError: 28382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533690 with bad export cookie 10609290483794180605 [ 483.704465] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 483.707142] LustreError: 28382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 483.892801] Lustre: server umount lustre-MDT0001 complete [ 488.632856] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 490.024781] Key type lgssc unregistered [ 490.187369] LNet: 31958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 490.191532] LNetError: 31958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 490.201653] LNet: Removed LNI 192.168.203.103@tcp [ 490.643151] Key type .llcrypt unregistered [ 490.645282] Key type ._llcrypt unregistered [ 502.351657] Key type ._llcrypt registered [ 502.353291] Key type .llcrypt registered [ 502.397666] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 502.902940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 502.921541] alg: No test for adler32 (adler32-zlib) [ 503.831871] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 503.951926] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 505.551103] Key type lgssc registered [ 506.077946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 509.717120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 509.729141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.968179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 512.759810] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 516.839106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.083773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 519.423422] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 521.945283] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 523.593766] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 526.874512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 527.085343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 528.103780] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 529.627757] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 532.203409] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 543.029647] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 20:42:30 (1780533750) [ 543.823853] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 544.588552] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 547.287497] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 20:42:34 (1780533754) [ 550.749669] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 20:42:37 (1780533757) [ 551.206811] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 554.889280] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 554.892767] Lustre: Skipped 2 previous similar messages [ 555.686611] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 20:42:42 (1780533762) [ 559.693126] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 559.697827] Lustre: Skipped 4 previous similar messages [ 567.079505] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 20:42:53 (1780533773) [ 568.519626] Lustre: *** cfs_fail_loc=131b, val=2*** [ 570.354386] Lustre: *** cfs_fail_loc=131b, val=2*** [ 574.685339] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 20:43:01 (1780533781) [ 575.637955] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 575.641437] Lustre: Skipped 9 previous similar messages [ 582.728877] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 20:43:09 (1780533789) [ 589.793418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 589.801684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.809576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 592.869074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 592.871443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.872657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 595.178821] Lustre: server umount lustre-OST0000 complete [ 597.985924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.998101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 598.001307] Lustre: Skipped 1 previous similar message [ 603.105193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 603.110206] Lustre: Skipped 1 previous similar message [ 608.225019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 608.228654] Lustre: Skipped 1 previous similar message [ 611.295080] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 611.372467] Lustre: server umount lustre-MDT0000 complete [ 613.067335] LustreError: 33313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533819 with bad export cookie 4210718405491007856 [ 613.070225] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.076455] LustreError: 33313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 613.239656] Lustre: server umount lustre-MDT0001 complete [ 617.647323] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 618.957876] Key type lgssc unregistered [ 619.116530] LNet: 40813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 619.121748] LNetError: 40813:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 619.130323] LNet: Removed LNI 192.168.203.103@tcp [ 619.528408] Key type .llcrypt unregistered [ 619.532508] Key type ._llcrypt unregistered [ 627.292647] Key type ._llcrypt registered [ 627.294279] Key type .llcrypt registered [ 627.342149] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 635.109682] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 635.533817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 635.588656] alg: No test for adler32 (adler32-zlib) [ 636.485390] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 636.633180] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 638.239154] Key type lgssc registered [ 638.719266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 654.184037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 654.197671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.334675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 655.357956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 655.402742] Lustre: lustre-MDT0000: new disk, initializing [ 655.457343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 655.471559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 657.329175] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 662.660543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.710129] Lustre: 43907:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 662.739386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 662.742074] Lustre: Skipped 1 previous similar message [ 662.783238] Lustre: lustre-MDT0001: new disk, initializing [ 662.825422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 662.841620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 662.846046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 664.408522] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 668.240249] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 669.757887] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 672.821985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 672.949315] Lustre: lustre-OST0000: new disk, initializing [ 672.951384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 672.954502] Lustre: 45651:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 672.995649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 675.334920] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 675.520456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 675.524978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 675.577044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 679.153496] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 680.923563] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 681.025591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 682.789678] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 682.877503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 686.052351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 686.056045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 686.064732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 686.068680] Lustre: Skipped 1 previous similar message [ 689.759534] Lustre: server umount lustre-OST0000 complete [ 693.728225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 693.731653] LustreError: Skipped 1 previous similar message [ 693.734403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 693.740244] Lustre: Skipped 1 previous similar message [ 693.743274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 696.289162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 696.289801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 696.298600] Lustre: Skipped 1 previous similar message [ 697.573077] Lustre: server umount lustre-MDT0000 complete [ 699.470441] LustreError: 43900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533905 with bad export cookie 2714060440911199911 [ 699.474947] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 699.475922] LustreError: 43900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 699.645918] Lustre: server umount lustre-MDT0001 complete [ 703.928781] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 20:45:10 (1780533910) [ 706.856755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.097985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 708.777420] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 712.161847] LustreError: 47962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.178609] LustreError: 47962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 712.963314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.177106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 714.980799] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 717.840701] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 719.596140] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 722.853155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 723.018108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 725.610825] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 728.086824] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 735.399432] LustreError: 50152:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 735.786725] Lustre: Modifying parameter general.timeout=1 in log params [ 736.925477] Lustre: Modifying parameter general.timeout=4 in log params [ 736.927983] Lustre: Skipped 2 previous similar messages [ 739.323512] Lustre: Modifying parameter general.timeout=10 in log params [ 739.326501] Lustre: Skipped 5 previous similar messages [ 743.336776] Lustre: Modifying parameter general.timeout=19 in log params [ 743.339785] Lustre: Skipped 8 previous similar messages [ 751.732605] Lustre: Modifying parameter general.timeout=41 in log params [ 751.735455] Lustre: Skipped 21 previous similar messages [ 767.806369] Lustre: Modifying parameter general.timeout=83 in log params [ 767.811822] Lustre: Skipped 41 previous similar messages [ 789.165482] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 20:46:36 (1780533996) [ 790.368055] Lustre: 57066:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 791.999088] Lustre: 57262:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 792.004565] Lustre: 57262:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 794.225761] Lustre: 57558:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 794.232277] Lustre: 57558:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 799.972381] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 799.979371] Lustre: Skipped 63 previous similar messages [ 805.141941] Lustre: 58903:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 805.148064] LustreError: 58903:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 806.935076] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 20:46:53 (1780534013) [ 815.584306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 815.590411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 815.597871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 817.900324] Lustre: server umount lustre-MDT0000 complete [ 819.562331] LustreError: 49578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534025 with bad export cookie 2714060440911200793 [ 819.566903] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 819.569135] LustreError: 49578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 819.701606] Lustre: server umount lustre-MDT0001 complete [ 831.533853] Lustre: server umount lustre-OST0000 complete [ 843.343425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 843.503362] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 843.672045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 845.411968] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 849.124913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.188828] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 849.204763] Lustre: 61427:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 851.186786] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 853.606449] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 855.270900] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 858.188522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 858.280593] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 861.048760] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 863.407755] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 869.638475] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 869.645218] Lustre: Skipped 6 previous similar messages [ 869.664179] Lustre: 63654:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 872.928995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 872.929597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.933053] LustreError: Skipped 1 previous similar message [ 872.940097] Lustre: Skipped 2 previous similar messages [ 872.942776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 872.947857] Lustre: Skipped 4 previous similar messages [ 874.196684] Lustre: server umount lustre-OST0000 complete [ 877.924496] Lustre: server umount lustre-MDT0000 complete [ 878.048859] LustreError: 61432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 878.057450] LustreError: 61432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 879.685098] LustreError: 63063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534086 with bad export cookie 2714060440911203194 [ 879.687835] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 879.690863] LustreError: 63063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 879.830314] Lustre: server umount lustre-MDT0001 complete [ 884.499562] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 885.770086] Key type lgssc unregistered [ 885.923396] LNet: 64839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.928057] LNetError: 64839:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.940343] LNet: Removed LNI 192.168.203.103@tcp [ 886.288147] Key type .llcrypt unregistered [ 886.289895] Key type ._llcrypt unregistered [ 897.360764] Key type ._llcrypt registered [ 897.362186] Key type .llcrypt registered [ 897.410301] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 897.813786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 897.872953] alg: No test for adler32 (adler32-zlib) [ 898.742647] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 898.839682] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 900.431199] Key type lgssc registered [ 900.878790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 904.023253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 904.032925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 905.291202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.941400] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 910.570175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.853406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 912.450781] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 914.572099] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 916.000613] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 918.629433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 918.866857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 921.169970] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 923.446612] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 937.210189] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 940.513553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 940.514383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 940.518620] Lustre: Skipped 2 previous similar messages [ 940.520902] Lustre: Skipped 1 previous similar message [ 945.632436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 945.635026] Lustre: Skipped 2 previous similar messages [ 946.076787] Lustre: server umount lustre-MDT0000 complete [ 947.494885] LustreError: 66194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534153 with bad export cookie 5093045812456400084 [ 947.498451] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 947.501055] LustreError: 66194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 947.616810] Lustre: server umount lustre-MDT0001 complete [ 959.321141] Lustre: server umount lustre-OST0000 complete [ 969.103682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 969.212949] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 969.232862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 969.348195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 970.919871] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 974.213552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.253912] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 974.265389] Lustre: 70766:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 974.455488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 975.927480] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 977.899933] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 979.273250] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 982.044895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 982.142590] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 984.607816] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 986.717629] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 987.379793] Lustre: 73001:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 988.069258] Lustre: Modifying parameter general.jobid_var=disable in log params [ 994.273813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 994.274223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.277219] LustreError: Skipped 1 previous similar message [ 994.277918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 994.284530] Lustre: Skipped 1 previous similar message [ 994.293361] Lustre: Skipped 1 previous similar message [ 995.567132] Lustre: server umount lustre-OST0000 complete [ 999.394896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.395670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 999.399926] Lustre: Skipped 1 previous similar message [ 1003.294640] Lustre: server umount lustre-MDT0000 complete [ 1004.512103] LustreError: 70776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.519978] LustreError: 70776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1004.695246] LustreError: 70758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534211 with bad export cookie 5093045812456400910 [ 1004.698215] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1004.699648] LustreError: 70758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1004.819868] Lustre: server umount lustre-MDT0001 complete [ 1008.698000] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1009.810924] Key type lgssc unregistered [ 1009.944535] LNet: 74194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.948039] LNetError: 74194:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.956328] LNet: Removed LNI 192.168.203.103@tcp [ 1010.263665] Key type .llcrypt unregistered [ 1010.265364] Key type ._llcrypt unregistered [ 1019.638303] Key type ._llcrypt registered [ 1019.639343] Key type .llcrypt registered [ 1019.674703] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1019.980253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1020.019042] alg: No test for adler32 (adler32-zlib) [ 1020.873089] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1020.950308] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1022.527174] Key type lgssc registered [ 1022.917561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1025.690027] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1025.696913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.867401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.262858] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1031.343455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.572814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1033.021982] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1034.909405] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1036.214122] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1038.491794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1038.685644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.583786] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1042.473571] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1055.167393] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1055.791262] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1055.793407] Lustre: Skipped 1 previous similar message [ 1057.030991] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1057.033236] Lustre: Skipped 3 previous similar messages [ 1059.360298] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1059.364091] Lustre: Skipped 6 previous similar messages [ 1063.565581] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1063.568713] Lustre: Skipped 12 previous similar messages [ 1071.838915] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1071.842110] Lustre: Skipped 24 previous similar messages [ 1090.308656] Lustre: 83315:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1090.628524] Lustre: Disabling parameter general.jobid_var= in log params [ 1090.630596] Lustre: Skipped 48 previous similar messages [ 1091.244433] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 20:51:38 (1780534298) [ 1091.822430] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1092.519290] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 20:51:39 (1780534299) [ 1097.183884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1097.186908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.194732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1098.211592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.211947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1098.213223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1098.225802] Lustre: Skipped 1 previous similar message [ 1099.729944] Lustre: server umount lustre-OST0000 complete [ 1103.332182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.333276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1103.338523] Lustre: Skipped 1 previous similar message [ 1103.343321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1107.354772] Lustre: server umount lustre-MDT0000 complete [ 1108.448967] LustreError: 75567:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1108.455961] LustreError: 75567:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1108.758189] LustreError: 77194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534315 with bad export cookie 3880373103571495325 [ 1108.762125] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1108.762903] LustreError: 77194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1108.891076] Lustre: server umount lustre-MDT0001 complete [ 1112.822405] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1114.006742] Key type lgssc unregistered [ 1114.164416] LNet: 84544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.170302] LNetError: 84544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.180674] LNet: Removed LNI 192.168.203.103@tcp [ 1114.525652] Key type .llcrypt unregistered [ 1114.527605] Key type ._llcrypt unregistered [ 1116.077993] Key type ._llcrypt registered [ 1116.079699] Key type .llcrypt registered [ 1116.122621] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1117.735085] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules [ 1118.231113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1118.239464] alg: No test for adler32 (adler32-zlib) [ 1119.113264] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1119.209935] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1120.815195] Key type lgssc registered [ 1121.447912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1121.938633] LustreError: 85666:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1129.658572] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1133.047968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1142.985824] LustreError: 85666:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1142.998871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1144.207100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.789598] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1149.132121] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 20:52:36 (1780534356) [ 1150.893275] LustreError: 86704:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1150.897459] LustreError: 86704:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1150.901015] LustreError: 86704:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1151.019517] Lustre: server umount lustre-MDT0000 complete [ 1156.092819] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1157.268464] Key type lgssc unregistered [ 1157.408503] LNet: 88164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.412308] LNetError: 88164:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1157.425561] LNet: Removed LNI 192.168.203.103@tcp [ 1157.712273] Key type .llcrypt unregistered [ 1157.713985] Key type ._llcrypt unregistered [ 1165.469057] Key type ._llcrypt registered [ 1165.470535] Key type .llcrypt registered [ 1165.512356] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1165.927661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1165.934191] alg: No test for adler32 (adler32-zlib) [ 1166.811909] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1166.900247] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1168.487151] Key type lgssc registered [ 1168.915274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1172.032051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1172.041918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1173.297730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1174.783103] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1178.006895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.271344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1179.679676] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1181.427889] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1182.658809] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1184.950115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1185.168134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1187.089651] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1188.946442] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1220.827218] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 20:53:47 (1780534427) [ 1221.371721] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1221.951118] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 20:53:49 (1780534429) [ 1224.160867] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1224.161991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1224.162186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1224.165016] Lustre: Skipped 1 previous similar message [ 1229.012830] Lustre: server umount lustre-MDT0000 complete [ 1229.281121] LustreError: 90670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1229.286617] LustreError: 90670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1230.192045] LustreError: 89322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534436 with bad export cookie 12768717919524370253 [ 1230.194899] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.197317] LustreError: 89322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1230.295948] Lustre: server umount lustre-MDT0001 complete [ 1240.823456] Lustre: server umount lustre-OST0000 complete [ 1244.591420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1244.735932] LustreError: 93295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.780231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1245.983752] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1248.582537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.158350] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1251.890170] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1253.195511] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1257.412121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1257.486209] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1257.489442] LustreError: 93283:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1257.500074] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1257.503656] LustreError: 95151:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1257.506388] LustreError: 95151:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1257.508600] LustreError: 95151:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1257.537588] Lustre: server umount lustre-OST0000 complete [ 1257.539279] LustreError: 95151:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1260.336964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1260.403138] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1260.407031] LustreError: 93284:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1260.416100] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1260.419666] LustreError: 95616:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1260.422180] LustreError: 95616:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1260.424240] LustreError: 95616:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1260.447031] Lustre: server umount lustre-OST0000 complete [ 1260.448602] LustreError: 95616:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1263.489753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1263.554252] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1263.556299] Lustre: Found index 0 for lustre-OST0000, updating log [ 1263.558224] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1263.580905] Lustre: lustre-OST0000: new disk, initializing [ 1263.582598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1263.584863] Lustre: 96147:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1263.668122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.671271] Lustre: Skipped 1 previous similar message [ 1265.548418] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1266.764786] Lustre: Failing over lustre-OST0000 [ 1266.811346] Lustre: server umount lustre-OST0000 complete [ 1268.704300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.704849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1268.707741] Lustre: Skipped 2 previous similar messages [ 1268.709531] Lustre: Skipped 2 previous similar messages [ 1272.800381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1272.802772] Lustre: Skipped 2 previous similar messages [ 1274.131200] Lustre: server umount lustre-MDT0000 complete [ 1275.397077] LustreError: 93276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534481 with bad export cookie 12768717919524372983 [ 1275.399580] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1275.401530] LustreError: 93276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1275.505369] Lustre: server umount lustre-MDT0001 complete [ 1278.170516] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 20:54:45 (1780534485) [ 1283.128805] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1286.607415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.761880] LustreError: 98453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.794880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1288.068060] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1290.808764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1292.388238] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1293.359896] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1295.731062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1297.982377] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1300.960331] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1300.995295] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1301.033070] Lustre: lustre-OST0001: new disk, initializing [ 1301.035529] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1301.038080] Lustre: 100719:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1302.459829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1302.464392] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1302.500305] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1303.054216] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1306.065762] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1311.734645] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1314.109050] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.3@tcp (stopping) [ 1316.831905] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1316.834283] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.130268] Lustre: server umount lustre-MDT0001 complete [ 1319.222724] LustreError: 98450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1322.169907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1322.207930] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1322.209817] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1322.224604] Lustre: 98441:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1322.485665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1322.488658] Lustre: Skipped 3 previous similar messages [ 1323.685770] LustreError: 102403:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1323.689545] LustreError: 102403:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1323.692070] LustreError: 98445:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1323.719867] LustreError: 102416:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.103@tcp: cfg command failed: rc = -17 [ 1323.723905] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1323.723905] [ 1323.804217] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1326.868812] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 20:55:34 (1780534534) [ 1327.591185] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1332.697903] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1338.919439] Lustre: 104277:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1346.689633] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1347.815848] Lustre: 105410:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1394.144159] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1394.144891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1394.145125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.145132] Lustre: Skipped 5 previous similar messages [ 1394.147857] Lustre: Skipped 4 previous similar messages [ 1398.563173] Lustre: server umount lustre-MDT0000 complete [ 1399.263970] LustreError: 100095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.268481] LustreError: 100095:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1399.750573] LustreError: 98434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534606 with bad export cookie 12768717919524373851 [ 1399.752138] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1399.755592] LustreError: 98434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1399.869556] Lustre: server umount lustre-MDT0001 complete [ 1411.598398] Lustre: server umount lustre-OST0000 complete [ 1423.196786] Lustre: server umount lustre-OST0001 complete [ 1424.965309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1427.886614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1430.420453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1433.442948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1438.990626] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1442.151305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1442.157591] Lustre: lustre-MDT0000: reset Object Index mappings [ 1442.293053] LustreError: 109315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1442.318129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1443.514828] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1446.424563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.434526] Lustre: lustre-MDT0001: reset Object Index mappings [ 1448.152301] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1449.163775] Lustre: 110429:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1451.423409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1452.710932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 1453.604482] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1455.782913] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 1456.254948] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1458.114707] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1461.147241] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1461.735992] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 1461.736819] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 1462.268662] Lustre: 112274:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1475.218835] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 20:58:02 (1780534682) [ 1507.811673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.813135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.817382] Lustre: Skipped 3 previous similar messages [ 1507.819655] Lustre: Skipped 4 previous similar messages [ 1511.007873] Lustre: server umount lustre-MDT0000 complete [ 1512.094403] LustreError: 110049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534718 with bad export cookie 12768717919524420611 [ 1512.097212] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1512.100078] LustreError: 110049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1512.928445] LustreError: 109312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1512.934706] LustreError: 109312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1518.456621] Lustre: server umount lustre-MDT0001 complete [ 1525.826984] Lustre: server umount lustre-OST0000 complete [ 1533.339275] Lustre: server umount lustre-OST0001 complete [ 1535.375058] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 1538.488907] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1549.711849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.799501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1549.812447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1549.850032] Lustre: lustre-MDT0000: new disk, initializing [ 1549.875290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1549.878926] Lustre: Skipped 3 previous similar messages [ 1549.885715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1551.160696] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1553.989954] Lustre: server umount lustre-MDT0000 complete [ 1556.831453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1556.918019] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1556.922439] Lustre: 116482:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1556.927487] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1558.215232] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1559.756628] Lustre: server umount lustre-MDT0000 complete [ 1562.478374] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 20:59:29 (1780534769) [ 1562.972279] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1563.522910] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 20:59:30 (1780534770) [ 1568.565970] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1571.663134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1573.036501] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1575.793032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1575.830859] Lustre: 118499:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1575.850222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1575.852247] Lustre: Skipped 1 previous similar message [ 1575.883078] Lustre: lustre-MDT0001: new disk, initializing [ 1575.921809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1575.925882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1577.115294] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1579.255758] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1581.319558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1581.426582] Lustre: lustre-OST0000: new disk, initializing [ 1581.429143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1581.432971] Lustre: 120043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1582.454680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1582.458482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1582.486742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1583.261055] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1587.184871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1587.246479] Lustre: 120898:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1589.055224] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1589.124924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1589.129106] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1589.174850] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1593.349269] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1594.491853] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1699.745741] Lustre: 127266:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1699.749773] Lustre: 127266:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1699.752111] Lustre: 127266:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1699.755066] Lustre: 127266:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1699.757876] Lustre: 127266:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1699.760097] Lustre: 127266:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1773.987501] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:03:00 (1780534980) [ 1785.823873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1785.826063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.829340] Lustre: Skipped 2 previous similar messages [ 1785.831145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.832859] Lustre: Skipped 5 previous similar messages [ 1791.686786] Lustre: server umount lustre-MDT0000 complete [ 1792.942363] LustreError: 121642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534999 with bad export cookie 12768717919524472698 [ 1792.945769] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1792.948430] LustreError: 121642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1793.124729] Lustre: server umount lustre-MDT0001 complete [ 1809.695157] Lustre: 88656:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780535000/real 1780535000] req@ffff92788c30ea00 x1867025619438080/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780535016 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1809.706588] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.709681] Lustre: Skipped 3 previous similar messages [ 1810.686877] Lustre: server umount lustre-OST0000 complete [ 1826.271132] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1826.348138] Lustre: server umount lustre-OST0001 complete [ 1828.509608] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 1831.840329] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1843.680278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.771104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1843.782250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1843.784106] Lustre: Skipped 1 previous similar message [ 1843.818180] Lustre: lustre-MDT0000: new disk, initializing [ 1843.819596] Lustre: Skipped 1 previous similar message [ 1843.843527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.846454] Lustre: Skipped 5 previous similar messages [ 1843.852675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1845.172270] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1849.417335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.460439] Lustre: 130246:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1849.567320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1850.977633] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1853.991790] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1855.188817] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1857.350209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1857.448179] Lustre: 131986:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1858.936509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1858.980398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1859.260491] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1862.153773] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1865.056513] Lustre: lustre-MDD0000: changelog on [ 2178.273661] Lustre: lustre-MDD0001: changelog off [ 2178.566731] Lustre: 132592:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2178.846614] Lustre: lustre-MDD0000: changelog off [ 2181.602444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2181.602594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.609531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2181.611323] Lustre: Skipped 5 previous similar messages [ 2186.449477] Lustre: server umount lustre-OST0000 complete [ 2193.856609] Lustre: server umount lustre-MDT0000 complete [ 2195.145124] LustreError: 130238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535401 with bad export cookie 12768717919525269872 [ 2195.148456] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2195.149496] LustreError: 130238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2195.273826] Lustre: server umount lustre-MDT0001 complete [ 2198.932345] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2200.000620] Key type lgssc unregistered [ 2200.131372] LNet: 139934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.135191] LNetError: 139934:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2200.142311] LNet: Removed LNI 192.168.203.103@tcp [ 2200.457110] Key type .llcrypt unregistered [ 2200.458818] Key type ._llcrypt unregistered [ 2206.187000] Key type ._llcrypt registered [ 2206.188470] Key type .llcrypt registered [ 2206.223947] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 2211.876796] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2212.179794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2212.186608] alg: No test for adler32 (adler32-zlib) [ 2213.025526] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 2213.099879] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2214.679150] Key type lgssc registered [ 2215.038536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2225.118139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2225.123421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2226.202242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2226.211822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2226.243923] Lustre: lustre-MDT0000: new disk, initializing [ 2226.263712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2226.269411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2227.418192] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2231.291547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2231.324574] Lustre: 143075:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2231.345526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2231.347307] Lustre: Skipped 1 previous similar message [ 2231.377210] Lustre: lustre-MDT0001: new disk, initializing [ 2231.405367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2231.417730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2231.420919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2232.599926] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2235.549145] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2236.590902] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2238.655443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2238.750209] Lustre: lustre-OST0000: new disk, initializing [ 2238.752183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2238.754678] Lustre: 144821:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2238.782241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2240.418599] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2240.695230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2240.698499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2240.732400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2243.262488] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2245.309528] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_module obdecho/obdecho [ 2245.868486] LustreError: 145696:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2245.871761] LustreError: 145696:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 2246.404971] LustreError: 145792:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 2251.234942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.237322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2251.239032] Lustre: Skipped 2 previous similar messages [ 2251.241301] Lustre: Skipped 1 previous similar message [ 2255.511219] Lustre: server umount lustre-MDT0000 complete [ 2256.645993] LustreError: 144820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535462 with bad export cookie 7182284375436642956 [ 2256.649784] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.650245] LustreError: 144820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2256.752611] Lustre: server umount lustre-MDT0001 complete [ 2268.179863] Lustre: server umount lustre-OST0000 complete [ 2274.443333] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 2277.259808] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2287.840599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.917504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2287.927721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2287.966275] Lustre: lustre-MDT0000: new disk, initializing [ 2287.987985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2287.994518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2289.194155] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2293.064383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2293.103878] Lustre: 149468:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2293.128053] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2293.130557] Lustre: Skipped 1 previous similar message [ 2293.167972] Lustre: lustre-MDT0001: new disk, initializing [ 2293.195385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2293.205497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2293.208602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2294.311550] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2297.108532] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2298.170478] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2300.181269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2300.265129] Lustre: 151209:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2301.924421] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2303.217738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2303.220442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2303.246609] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2304.717223] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2305.888204] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2305.951033] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2307.111464] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2307.173784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2308.576915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2308.577705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.579504] LustreError: Skipped 1 previous similar message [ 2308.583754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2308.588211] Lustre: Skipped 1 previous similar message [ 2313.697904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2313.700121] Lustre: Skipped 1 previous similar message [ 2313.875202] Lustre: server umount lustre-OST0000 complete [ 2318.816886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2318.817078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2318.822581] Lustre: Skipped 1 previous similar message [ 2318.824372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2318.826208] Lustre: Skipped 1 previous similar message [ 2321.104710] Lustre: server umount lustre-MDT0000 complete [ 2322.194967] LustreError: 149460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535528 with bad export cookie 7182284375436643901 [ 2322.197962] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2322.198732] LustreError: 149460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2322.301616] Lustre: server umount lustre-MDT0001 complete [ 2323.340598] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:12:10 (1780535530) [ 2323.792378] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2324.296441] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:12:11 (1780535531) [ 2326.700308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.831707] LustreError: 153472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.838526] LustreError: 153472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2326.853682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2326.856293] Lustre: Skipped 1 previous similar message [ 2327.981092] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2330.593658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.911125] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2333.528484] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2334.604420] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2336.696889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2336.740495] LustreError: 155068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.602315] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2340.278805] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2350.907587] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.3@tcp (stopping) [ 2351.073810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.074908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2351.078828] Lustre: Skipped 3 previous similar messages [ 2353.363307] Lustre: server umount lustre-MDT0000 complete [ 2354.498294] LustreError: 153453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535560 with bad export cookie 7182284375436644769 [ 2354.501095] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2354.501716] LustreError: 153453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2356.021447] LustreError: 153497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.192211] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.199722] LustreError: 153467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.200037] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2359.205628] LustreError: 153467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2359.207694] Lustre: Skipped 5 previous similar messages [ 2360.688765] Lustre: server umount lustre-MDT0001 complete [ 2362.723153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2366.649776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2366.719283] LustreError: 156973:0:(llog.c:1646:llog_backup()) MGC192.168.203.103@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2366.723111] Lustre: 156973:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.103@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2376.373609] LustreError: 156978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.741590] LustreError: 156996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.748234] LustreError: 156996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2404.319989] LustreError: 156980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.325311] LustreError: 156980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 2427.487227] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2427.489128] Lustre: 156973:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.103@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2436.917697] LustreError: 156979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2436.923939] LustreError: 156979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 2468.326369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2468.330181] Lustre: Skipped 2 previous similar messages [ 2469.571364] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2472.350530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2473.442832] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2474.399109] LustreError: 157698:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2474.401948] LustreError: 157698:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000008fb99e3d x1867026707603840/t0(0) o250->MGC192.168.203.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1780535680 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2474.407943] LustreError: 157698:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2474.463566] LustreError: 141302:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000051659eee x1867026707607936/t0(0) o250->MGC192.168.203.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2475.866579] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2477.624295] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2478.135897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2479.236457] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2479.589194] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2482.842227] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2489.110565] Lustre: server umount lustre-MDT0000 complete [ 2489.825546] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.826441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2489.829091] Lustre: Skipped 1 previous similar message [ 2490.284946] LustreError: 153455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535696 with bad export cookie 7182284375436651209 [ 2490.288059] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2490.288569] LustreError: 153455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2490.406899] Lustre: server umount lustre-MDT0001 complete [ 2500.820945] Lustre: server umount lustre-OST0000 complete [ 2507.277751] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 2510.308299] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2522.346914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.443748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2522.455880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2522.457775] Lustre: Skipped 1 previous similar message [ 2522.490577] Lustre: lustre-MDT0000: new disk, initializing [ 2522.491900] Lustre: Skipped 1 previous similar message [ 2522.516826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.518877] Lustre: Skipped 1 previous similar message [ 2522.524920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2523.848193] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2528.028779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.075794] Lustre: 161785:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2528.178871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2529.465748] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2532.453588] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2533.646660] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2535.799454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2535.900656] Lustre: 163531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2537.667130] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2540.696141] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2541.047391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2541.081680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2542.010472] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2542.085952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2543.421182] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2543.493025] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2546.146795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.146796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2546.148169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2546.150373] Lustre: Skipped 2 previous similar messages [ 2550.227515] Lustre: server umount lustre-OST0000 complete [ 2557.592604] Lustre: server umount lustre-MDT0000 complete [ 2558.800278] LustreError: 161777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535765 with bad export cookie 7182284375436668002 [ 2558.803730] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.806026] LustreError: 161777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2558.930679] Lustre: server umount lustre-MDT0001 complete [ 2561.536251] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:16:08 (1780535768) [ 2563.587243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.736503] LustreError: 165794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.742545] LustreError: 165794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 31 previous similar messages [ 2564.998192] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2567.678189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.996916] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2570.655074] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2571.782431] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2573.896927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.707433] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2577.404959] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2585.253941] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:16:32 (1780535792) [ 2588.642533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2588.644032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2588.644615] LustreError: Skipped 2 previous similar messages [ 2588.648243] Lustre: Skipped 2 previous similar messages [ 2588.652127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2588.653880] Lustre: Skipped 5 previous similar messages [ 2592.336902] Lustre: server umount lustre-OST0000 complete [ 2599.696280] Lustre: server umount lustre-MDT0000 complete [ 2600.919075] LustreError: 167411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535807 with bad export cookie 7182284375436668884 [ 2600.922287] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2600.922520] LustreError: 167411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2601.033125] Lustre: server umount lustre-MDT0001 complete [ 2604.480380] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2605.400171] Key type lgssc unregistered [ 2605.525262] LNet: 169167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2605.527549] LNetError: 169167:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2605.533371] LNet: Removed LNI 192.168.203.103@tcp [ 2605.800706] Key type .llcrypt unregistered [ 2605.802095] Key type ._llcrypt unregistered [ 2612.659038] Key type ._llcrypt registered [ 2612.660119] Key type .llcrypt registered [ 2612.694259] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2613.006945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2613.065521] alg: No test for adler32 (adler32-zlib) [ 2613.911589] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 2613.987618] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2615.567110] Key type lgssc registered [ 2615.918557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2618.283953] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2618.290960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2633.759242] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2638.879202] LustreError: 170372:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.103@tcp: failed processing log, type 4: rc = -110 [ 2643.999396] LustreError: 170372:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2644.003207] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2644.004986] LustreError: MGC192.168.203.103@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2644.009180] LustreError: 170372:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2644.011880] LustreError: 170372:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 2644.014172] LustreError: 170372:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2644.036733] Lustre: server umount lustre-OST0000 complete [ 2644.038949] LustreError: 170372:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2646.693992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2647.841833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.046546] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2651.721054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.842740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2653.094357] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2654.773464] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2655.901301] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2658.047576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2658.111756] LustreError: 172461:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2658.115146] Lustre: 172461:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.103@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2658.156529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2659.922649] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2661.563534] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2662.367703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2662.369994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2662.373579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2663.394039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2663.394070] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.366027] Lustre: server umount lustre-MDT0000 complete [ 2668.512070] LustreError: 170872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.519874] LustreError: 170872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2669.521251] LustreError: 170853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535875 with bad export cookie 12159403481531668368 [ 2669.524916] LustreError: 170853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2669.525130] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2669.535128] LustreError: 171585:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000029be21a4 x1867027128012288/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2669.633213] Lustre: server umount lustre-MDT0001 complete [ 2681.112792] Lustre: server umount lustre-OST0000 complete [ 2683.858826] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 2686.648786] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2697.324498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2697.408690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2697.418714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2697.452712] Lustre: lustre-MDT0000: new disk, initializing [ 2697.473848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2697.479445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2698.615085] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2702.479112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.517287] Lustre: 175836:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2702.538229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2702.539994] Lustre: Skipped 1 previous similar message [ 2702.572641] Lustre: lustre-MDT0001: new disk, initializing [ 2702.602968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2702.613417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2702.616743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2703.751819] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2706.594792] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2707.675707] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2709.736519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2709.832092] Lustre: lustre-OST0000: new disk, initializing [ 2709.833778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2709.836358] Lustre: 177580:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2711.566279] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2714.618314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2714.621792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2714.651906] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2714.660328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2716.034358] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2716.100100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2717.462882] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2717.528277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2719.713615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2719.714577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.717513] LustreError: Skipped 1 previous similar message [ 2719.720961] Lustre: Skipped 1 previous similar message [ 2719.721555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2719.726427] Lustre: Skipped 3 previous similar messages [ 2720.202941] Lustre: server umount lustre-OST0000 complete [ 2723.295755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2723.298453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2723.302723] Lustre: Skipped 1 previous similar message [ 2723.304650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2727.511812] Lustre: server umount lustre-MDT0000 complete [ 2728.675340] LustreError: 177590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535935 with bad export cookie 12159403481531669061 [ 2728.678694] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2728.681162] LustreError: 177590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2728.793868] Lustre: server umount lustre-MDT0001 complete [ 2731.390623] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 21:18:58 (1780535938) [ 2736.573172] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2737.509573] Key type lgssc unregistered [ 2737.633340] LNet: 180362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2737.635864] LNetError: 180362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2737.643274] LNet: Removed LNI 192.168.203.103@tcp [ 2737.925606] Key type .llcrypt unregistered [ 2737.926993] Key type ._llcrypt unregistered [ 2744.677098] Key type ._llcrypt registered [ 2744.678226] Key type .llcrypt registered [ 2744.718398] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2745.044731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2745.053306] alg: No test for adler32 (adler32-zlib) [ 2745.896505] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 2745.979431] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2747.559129] Key type lgssc registered [ 2747.899128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2750.281630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2750.287868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.434514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2752.703614] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2755.395958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.510944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2756.713442] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2758.358847] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2759.499075] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2761.815966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2761.883975] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2761.885266] LustreError: 183176:0:(llog.c:1687:llog_backup()) MGC192.168.203.103@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2761.887898] Lustre: 183176:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.103@tcp: can't backup local config lustre-OST0000: rc = -28 [ 2761.922927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2763.653116] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2765.314677] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2770.400646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.403400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2770.406543] Lustre: Skipped 2 previous similar messages [ 2770.408341] Lustre: Skipped 1 previous similar message [ 2772.113978] Lustre: server umount lustre-MDT0000 complete [ 2773.263207] LustreError: 181519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535979 with bad export cookie 13723387822270960670 [ 2773.266644] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2773.268271] LustreError: 181519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2773.366331] Lustre: server umount lustre-MDT0001 complete [ 2784.722114] Lustre: server umount lustre-OST0000 complete [ 2787.642608] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 2790.565241] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2801.930606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.067913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2802.085560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2802.138267] Lustre: lustre-MDT0000: new disk, initializing [ 2802.173663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.183791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2803.514044] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2808.835460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2808.908493] Lustre: 186549:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2808.944467] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2808.947451] Lustre: Skipped 1 previous similar message [ 2808.950475] LustreError: 187317:0:(llog.c:1687:llog_backup()) MGC192.168.203.103@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 2808.957271] LustreError: 187317:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 2808.962270] Lustre: 187317:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.103@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 2808.977284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2808.981332] Lustre: Skipped 1 previous similar message [ 2809.027276] Lustre: lustre-MDT0001: new disk, initializing [ 2809.074751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2809.094609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2809.101263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2810.490316] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2814.273601] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2815.868719] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2819.145495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2819.298737] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2819.301649] Lustre: Skipped 1 previous similar message [ 2819.304734] LustreError: 188271:0:(llog.c:1687:llog_backup()) MGC192.168.203.103@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2819.312715] LustreError: 188271:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 2819.317928] Lustre: 188271:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.103@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 2819.325192] Lustre: 188271:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2819.339973] Lustre: lustre-OST0000: new disk, initializing [ 2819.342879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2819.347233] Lustre: 188295:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2819.387703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2822.011444] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2826.164903] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2826.763844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2826.769781] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2826.820795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2827.965313] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2828.072558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2829.849986] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2829.918355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2831.840597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2831.842244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.843314] LustreError: Skipped 1 previous similar message [ 2831.848129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2831.851375] Lustre: Skipped 2 previous similar messages [ 2836.620333] Lustre: server umount lustre-OST0000 complete [ 2840.031730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2840.034075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.037846] Lustre: Skipped 1 previous similar message [ 2840.039500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2842.081428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2842.082475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.200666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2847.203538] Lustre: Skipped 2 previous similar messages [ 2852.319195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2852.402189] Lustre: server umount lustre-MDT0000 complete [ 2854.001260] LustreError: 187331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536060 with bad export cookie 13723387822270961356 [ 2854.004547] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2854.005163] LustreError: 187331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2854.123480] Lustre: server umount lustre-MDT0001 complete [ 2856.948072] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 21:21:04 (1780536064) [ 2862.130781] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2865.293165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.426516] LustreError: 191255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2865.432741] LustreError: 191255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2865.439631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.691786] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2869.408343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2870.741137] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2871.624909] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2873.686293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2875.589253] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2878.267027] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2878.332163] Lustre: lustre-OST0001: new disk, initializing [ 2878.334267] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2878.337319] Lustre: 193508:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2880.084961] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2882.368145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2882.372958] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2882.400548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2885.351162] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2890.906351] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2891.464758] Lustre: DEBUG MARKER: ADD OST3 [ 2893.029849] loop: module loaded [ 2893.046512] loop0: detected capacity change from 0 to 204800000 [ 2893.126264] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2893.130524] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2893.135176] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2895.163861] loop0: detected capacity change from 0 to 204800000 [ 2896.918575] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2896.984053] Lustre: lustre-OST0003: new disk, initializing [ 2896.986364] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 2896.988701] Lustre: 195222:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2898.033948] Lustre: DEBUG MARKER: STOP OST3 [ 2898.571718] Lustre: *** cfs_fail_loc=2109, val=2*** [ 2898.609993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 2898.612865] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 2898.622362] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 2898.657711] Lustre: Failing over lustre-OST0003 [ 2898.703470] Lustre: server umount lustre-OST0003 complete [ 2898.934481] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2900.576119] LustreError: 195609:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 2902.844309] Lustre: DEBUG MARKER: START OST3 again [ 2903.357376] LustreError: 192704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.032651] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2904.034773] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.928501] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2905.000957] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 2905.003242] Lustre: Skipped 4 previous similar messages [ 2905.007666] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 2906.425600] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2906.473530] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2906.473545] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2906.730491] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2919.894292] Lustre: Failing over lustre-OST0003 [ 2919.937096] Lustre: server umount lustre-OST0003 complete [ 2920.421439] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.422789] LustreError: 192704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2920.425270] Lustre: Skipped 2 previous similar messages [ 2920.430953] LustreError: 192704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2925.539679] LustreError: 192704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.543976] LustreError: 192704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2930.660516] LustreError: 192705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2930.668271] LustreError: 192705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2935.781625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2935.783399] Lustre: Skipped 5 previous similar messages [ 2938.331290] Lustre: server umount lustre-MDT0000 complete [ 2939.484132] LustreError: 191281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536145 with bad export cookie 13723387822270962224 [ 2939.488258] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2939.488703] LustreError: 191281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2940.896126] LustreError: 191250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.896839] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.900874] LustreError: 191250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2940.904383] Lustre: Skipped 5 previous similar messages [ 2945.710667] Lustre: server umount lustre-MDT0001 complete [ 2946.987904] Lustre: server umount lustre-OST0000 complete [ 2948.215870] Lustre: server umount lustre-OST0001 complete [ 2950.096407] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 2952.366479] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2963.792073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.876558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2963.886468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2963.921178] Lustre: lustre-MDT0000: new disk, initializing [ 2963.947038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2965.142675] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2969.053534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2969.096905] Lustre: 200569:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2969.180728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2969.183908] Lustre: Skipped 1 previous similar message [ 2969.195598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2970.429251] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2973.365104] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2974.471711] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2976.567091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2976.684904] Lustre: 202314:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2978.184048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2978.225225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2978.534232] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2981.480387] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2982.686950] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2982.747306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2983.899013] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2983.960421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2988.513688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2988.514588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.516121] LustreError: Skipped 1 previous similar message [ 2988.520288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2988.522962] Lustre: Skipped 5 previous similar messages [ 2990.669465] Lustre: server umount lustre-OST0000 complete [ 2997.980270] Lustre: server umount lustre-MDT0000 complete [ 2998.752441] LustreError: 200580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2998.756568] LustreError: 200580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2999.155401] LustreError: 201348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536205 with bad export cookie 13723387822270964506 [ 2999.158515] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2999.159846] LustreError: 201348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2999.271861] Lustre: server umount lustre-MDT0001 complete [ 3000.499379] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:23:27 (1780536207) [ 3006.003712] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3009.261930] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3021.646080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3021.757194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3021.768933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3021.770546] Lustre: Skipped 3 previous similar messages [ 3021.814743] Lustre: lustre-MDT0000: new disk, initializing [ 3021.817218] Lustre: Skipped 2 previous similar messages [ 3021.850873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3021.853737] Lustre: Skipped 2 previous similar messages [ 3023.200270] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3027.348156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3027.391079] Lustre: 206747:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3027.493797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3028.758414] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3031.866349] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3033.034819] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3035.169785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3035.275802] Lustre: 208491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3036.442605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3037.056047] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3040.068903] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3041.383861] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3041.451886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3042.659604] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3042.723733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3046.880679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3046.883021] LustreError: Skipped 1 previous similar message [ 3046.883811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3046.886019] Lustre: Skipped 3 previous similar messages [ 3049.419815] Lustre: server umount lustre-OST0000 complete [ 3056.729953] Lustre: server umount lustre-MDT0000 complete [ 3057.120128] LustreError: 206760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.124980] LustreError: 206760:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3057.909567] LustreError: 206741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536264 with bad export cookie 13723387822270965388 [ 3057.912593] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3057.913220] LustreError: 206741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3058.026922] Lustre: server umount lustre-MDT0001 complete [ 3060.677372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.042225] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3064.619568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.017733] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3067.863985] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3069.169720] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3071.456189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3073.384332] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3075.069056] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3096.544735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3096.546983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3096.550214] Lustre: Skipped 8 previous similar messages [ 3106.271123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3106.322737] Lustre: server umount lustre-OST0000 complete [ 3111.904201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3111.906659] Lustre: Skipped 8 previous similar messages [ 3113.624944] Lustre: server umount lustre-MDT0000 complete [ 3114.771037] LustreError: 210542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536321 with bad export cookie 13723387822270966270 [ 3114.773752] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3114.775987] LustreError: 210542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3114.898287] Lustre: server umount lustre-MDT0001 complete [ 3206.977674] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 3208.437745] Key type lgssc unregistered [ 3208.597635] LNet: 213894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3208.603507] LNetError: 213894:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3208.612592] LNet: Removed LNI 192.168.203.103@tcp [ 3209.070138] Key type .llcrypt unregistered [ 3209.071587] Key type ._llcrypt unregistered [ 3218.460855] Key type ._llcrypt registered [ 3218.461882] Key type .llcrypt registered [ 3218.499880] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3224.736921] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3225.137226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3225.145739] alg: No test for adler32 (adler32-zlib) [ 3225.989428] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3226.068503] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 3227.655141] Key type lgssc registered [ 3228.055413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3240.882395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3240.896364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3242.000433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3242.013427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3242.047483] Lustre: lustre-MDT0000: new disk, initializing [ 3242.072343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3242.081344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3243.733594] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3249.071141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3249.146846] Lustre: 217186:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3249.185698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3249.190039] Lustre: Skipped 1 previous similar message [ 3249.241051] Lustre: lustre-MDT0001: new disk, initializing [ 3249.286240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3249.305207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3249.311929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3250.858259] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3253.953279] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3255.224845] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3258.050664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3258.156663] Lustre: lustre-OST0000: new disk, initializing [ 3258.158382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3258.160830] Lustre: 218934:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3258.186470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3260.025903] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3262.940560] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3264.217340] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3268.088977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3268.093480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3268.120841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3268.370390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3269.638087] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3269.727840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3273.189847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3273.190810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.193952] LustreError: Skipped 1 previous similar message [ 3273.202857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3273.208410] Lustre: Skipped 1 previous similar message [ 3276.558039] Lustre: server umount lustre-OST0000 complete [ 3278.305420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.306497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3278.313752] Lustre: Skipped 2 previous similar messages [ 3278.323313] Lustre: Skipped 1 previous similar message [ 3283.424087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.426297] Lustre: Skipped 1 previous similar message [ 3284.312230] Lustre: server umount lustre-MDT0000 complete [ 3285.539639] LustreError: 217180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536491 with bad export cookie 14221951489942001160 [ 3285.543651] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3285.545383] LustreError: 217180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3285.672484] Lustre: server umount lustre-MDT0001 complete [ 3292.544902] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3295.726668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3295.872956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3297.072570] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3299.838967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3300.002124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3301.557320] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3302.551980] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3305.633694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3308.114126] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3310.845093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3310.907712] Lustre: lustre-OST0001: new disk, initializing [ 3310.909410] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3310.911986] Lustre: 223989:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3310.945285] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3310.948343] Lustre: Skipped 1 previous similar message [ 3312.073382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3312.076825] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3312.104525] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3312.744367] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3318.610056] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3324.542618] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3328.513022] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:28:55 (1780536535) [ 3331.040391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3331.044978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.051793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3335.584405] Lustre: server umount lustre-MDT0000 complete [ 3337.045509] LustreError: 221715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536543 with bad export cookie 14221951489942002042 [ 3337.050243] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3337.050252] LustreError: 221715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3337.180473] Lustre: server umount lustre-MDT0001 complete [ 3348.609501] Lustre: server umount lustre-OST0000 complete [ 3360.304662] Lustre: server umount lustre-OST0001 complete [ 3362.446411] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3366.293970] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3381.319786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3381.462389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3381.480711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3381.537304] Lustre: lustre-MDT0000: new disk, initializing [ 3381.573516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.580721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3383.175853] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3388.235777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3388.292317] Lustre: 227881:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3388.438248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3390.090621] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3393.810533] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3395.185575] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3398.016371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3398.172282] Lustre: lustre-OST0000: new disk, initializing [ 3398.174630] Lustre: Skipped 1 previous similar message [ 3398.177169] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3398.180086] Lustre: Skipped 2 previous similar messages [ 3398.183124] Lustre: 229624:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3399.797933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3399.800973] Lustre: Skipped 1 previous similar message [ 3399.802850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3399.837352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3400.313375] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3404.100900] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3405.713216] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3405.784074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3407.274891] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3407.375412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3410.402312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3410.405273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.407719] LustreError: Skipped 1 previous similar message [ 3410.415439] Lustre: Skipped 3 previous similar messages [ 3410.416507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3410.425055] Lustre: Skipped 5 previous similar messages [ 3414.374609] Lustre: server umount lustre-OST0000 complete [ 3419.104504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3419.109370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.118378] Lustre: Skipped 1 previous similar message [ 3419.122522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3421.863365] Lustre: server umount lustre-MDT0000 complete [ 3423.594568] LustreError: 227873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536629 with bad export cookie 14221951489942004135 [ 3423.600438] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3423.602135] LustreError: 227873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3423.768314] Lustre: server umount lustre-MDT0001 complete [ 3427.023093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.236908] LustreError: 231693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3427.246959] LustreError: 231693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3427.274271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3427.278438] Lustre: Skipped 2 previous similar messages [ 3428.813697] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3431.676793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3433.172247] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3435.019786] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3436.662412] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3439.295538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3441.796676] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3443.820554] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3590.112523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3590.117812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3590.125806] Lustre: Skipped 1 previous similar message [ 3590.129559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3590.133567] Lustre: Skipped 2 previous similar messages [ 3595.475271] Lustre: server umount lustre-OST0000 complete [ 3602.774052] Lustre: server umount lustre-MDT0000 complete [ 3603.425114] LustreError: 231712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3603.429956] LustreError: 231712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3604.044137] LustreError: 231720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536810 with bad export cookie 14221951489942005017 [ 3604.048054] LustreError: 231720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3604.048135] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3604.162269] Lustre: server umount lustre-MDT0001 complete [ 3706.993553] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 3708.339701] Key type lgssc unregistered [ 3708.507599] LNet: 234949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3708.512948] LNetError: 234949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3708.525439] LNet: Removed LNI 192.168.203.103@tcp [ 3708.977104] Key type .llcrypt unregistered [ 3708.979079] Key type ._llcrypt unregistered [ 3719.252946] Key type ._llcrypt registered [ 3719.254284] Key type .llcrypt registered [ 3719.303513] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3726.167900] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3726.614358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3726.635346] alg: No test for adler32 (adler32-zlib) [ 3727.502287] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3727.609938] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 3729.207155] Key type lgssc registered [ 3729.755797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3748.177036] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3752.717688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3752.730778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3753.848574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3753.861075] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3753.906621] Lustre: lustre-MDT0000: new disk, initializing [ 3753.930862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3753.937260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3755.211606] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3759.757379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3759.800982] Lustre: 238940:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3759.829149] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3759.831588] Lustre: Skipped 1 previous similar message [ 3759.864983] Lustre: lustre-MDT0001: new disk, initializing [ 3759.898521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3759.909793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3759.914276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3761.405733] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3763.678836] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3766.534928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3766.633231] Lustre: lustre-OST0000: new disk, initializing [ 3766.634870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3766.637225] Lustre: 240540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3766.667444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3768.452593] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3769.995507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3770.002934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3770.074225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3773.023875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3773.099222] Lustre: lustre-OST0001: new disk, initializing [ 3773.101330] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3773.104197] Lustre: 241396:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3773.138956] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3775.075091] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3775.366849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3775.371353] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3775.409513] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3781.365899] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3785.746356] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3790.822188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3790.825430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3790.827856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3790.830941] Lustre: Skipped 2 previous similar messages [ 3790.834617] Lustre: Skipped 1 previous similar message [ 3795.598317] Lustre: server umount lustre-MDT0000 complete [ 3797.202707] LustreError: 238933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537003 with bad export cookie 11762759312275112713 [ 3797.207200] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3797.207269] LustreError: 238933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3797.324511] Lustre: server umount lustre-MDT0001 complete [ 3809.283224] Lustre: server umount lustre-OST0000 complete [ 3819.818857] Lustre: server umount lustre-OST0001 complete [ 3824.374272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3827.282319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3830.132400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3833.107027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3839.220651] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3842.400434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3842.407288] Lustre: lustre-MDT0000: reset Object Index mappings [ 3842.554439] LustreError: 246363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3842.561597] LustreError: 246363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3842.574310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3843.810492] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3846.611084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3846.617118] Lustre: lustre-MDT0001: reset Object Index mappings [ 3848.082474] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3849.046091] Lustre: 247473:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3852.308687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3852.471852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3852.474118] Lustre: Skipped 1 previous similar message [ 3854.565526] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3857.581863] LustreError: 248495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3858.700111] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3861.167420] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3867.117555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3867.118929] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 3870.289474] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3871.573279] Lustre: 249310:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3877.347613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3877.349744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3877.353227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3877.353240] Lustre: Skipped 2 previous similar messages [ 3877.355840] Lustre: Skipped 3 previous similar messages [ 3878.957577] Lustre: server umount lustre-MDT0000 complete [ 3880.411537] LustreError: 247090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537086 with bad export cookie 11762759312275119496 [ 3880.416296] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3880.416363] LustreError: 247090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3880.538713] Lustre: server umount lustre-MDT0001 complete [ 3892.694286] Lustre: server umount lustre-OST0000 complete [ 3904.571959] Lustre: server umount lustre-OST0001 complete [ 3912.455568] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:38:39 (1780537119) [ 3918.401943] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3922.067192] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3936.891292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3937.003346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3937.015994] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3937.051352] Lustre: lustre-MDT0000: new disk, initializing [ 3937.077502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3937.080772] Lustre: Skipped 1 previous similar message [ 3937.088391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3938.759512] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3943.687092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3943.724660] Lustre: 253528:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3943.828378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3945.291984] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3948.494074] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3949.938911] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3952.378564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3952.532034] Lustre: lustre-OST0000: new disk, initializing [ 3952.536356] Lustre: Skipped 1 previous similar message [ 3952.540210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3952.545661] Lustre: Skipped 2 previous similar messages [ 3952.549306] Lustre: 255271:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3953.855230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3953.861612] Lustre: Skipped 1 previous similar message [ 3953.864903] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3953.901796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3954.675045] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3958.064202] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3959.621227] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3959.713813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3961.138518] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3961.204514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3964.387893] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3964.389119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.393295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3964.398838] Lustre: Skipped 2 previous similar messages [ 3964.402063] Lustre: Skipped 3 previous similar messages [ 3969.509750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3969.514065] Lustre: Skipped 1 previous similar message [ 3974.626205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3974.629886] Lustre: Skipped 1 previous similar message [ 3976.159127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3976.219761] Lustre: server umount lustre-OST0000 complete [ 3979.746394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3979.746826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3979.753825] Lustre: Skipped 1 previous similar message [ 3979.763411] LustreError: Skipped 1 previous similar message [ 3983.676436] Lustre: server umount lustre-MDT0000 complete [ 3984.864295] LustreError: 253541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3984.876025] LustreError: 253541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3985.166288] LustreError: 253521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537191 with bad export cookie 11762759312275121568 [ 3985.172834] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3985.175101] LustreError: 253521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3985.313491] Lustre: server umount lustre-MDT0001 complete [ 3991.593650] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3995.488722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3995.688189] LustreError: 258042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3995.711893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.716641] Lustre: Skipped 2 previous similar messages [ 3997.190742] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4000.311130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.674464] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4002.682581] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4005.437609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4007.528925] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4011.110723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4011.170497] Lustre: lustre-OST0001: new disk, initializing [ 4011.172202] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4011.174521] Lustre: 260304:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4012.999687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4013.002581] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4013.035175] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4013.097224] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4017.429746] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4018.973914] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4020.771761] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4020.808299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4053.985725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4053.988219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4053.989764] Lustre: Skipped 3 previous similar messages [ 4053.991561] Lustre: Skipped 4 previous similar messages [ 4058.983555] Lustre: server umount lustre-MDT0000 complete [ 4059.106666] LustreError: 258043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4059.111527] LustreError: 258043:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4060.291576] LustreError: 258024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537266 with bad export cookie 11762759312275122450 [ 4060.296688] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4060.298443] LustreError: 258024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4060.430985] Lustre: server umount lustre-MDT0001 complete [ 4072.464971] Lustre: server umount lustre-OST0000 complete [ 4084.283436] Lustre: server umount lustre-OST0001 complete [ 4091.278744] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4095.506091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4095.707300] LustreError: 263355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4095.729161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4095.733887] Lustre: Skipped 3 previous similar messages [ 4097.356626] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4100.913030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4102.478252] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4103.718409] Lustre: 264463:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4106.917480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4109.609307] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4113.524121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4116.159532] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4121.069397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4121.075721] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4124.756570] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4126.657864] Lustre: 266303:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4129.090349] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4129.099348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4133.768931] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:42:20 (1780537340) [ 4162.019585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.025157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4162.033755] Lustre: Skipped 2 previous similar messages [ 4162.037839] Lustre: Skipped 1 previous similar message [ 4167.434227] Lustre: server umount lustre-MDT0000 complete [ 4169.091434] LustreError: 263336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537375 with bad export cookie 11762759312275129282 [ 4169.098786] LustreError: 263336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4169.099621] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4172.256680] LustreError: 266068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4172.268478] LustreError: 266068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4175.317171] Lustre: server umount lustre-MDT0001 complete [ 4183.177689] Lustre: server umount lustre-OST0000 complete [ 4199.391184] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4199.456584] Lustre: server umount lustre-OST0001 complete [ 4201.862247] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 4206.411340] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4227.414772] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4231.976776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4232.060183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4232.071076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4232.105567] Lustre: lustre-MDT0000: new disk, initializing [ 4232.127626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4232.129475] Lustre: Skipped 3 previous similar messages [ 4232.134623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4233.515093] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4238.881276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.949871] Lustre: 270493:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4239.121417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4240.890924] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4246.693322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4246.843640] Lustre: 272099:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4247.921686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4249.529953] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4254.437157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4255.953755] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4256.790888] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4261.821881] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4269.529645] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 21:44:36 (1780537476) [ 4271.591214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4271.593692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4271.601816] Lustre: Skipped 5 previous similar messages [ 4271.609856] Lustre: Skipped 11 previous similar messages [ 4276.711181] LustreError: 270500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4276.717952] LustreError: 270500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4276.730730] Lustre: server umount lustre-MDT0000 complete [ 4278.511049] LustreError: 270485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537484 with bad export cookie 11762759312275136716 [ 4278.517897] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4278.518755] LustreError: 270485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4278.669546] Lustre: server umount lustre-MDT0001 complete [ 4290.057111] Lustre: server umount lustre-OST0000 complete [ 4301.615569] Lustre: server umount lustre-OST0001 complete [ 4305.750662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.791351] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4311.895345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4313.804266] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4316.091087] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4317.396673] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4320.421545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4322.979437] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4324.879468] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4341.223174] LustreError: 276787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4341.234974] LustreError: 276787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 4342.336090] loop1: detected capacity change from 0 to 204800000 [ 4342.422044] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4342.425984] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4342.429959] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4344.450842] loop1: detected capacity change from 0 to 204800000 [ 4346.912656] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4346.962799] LustreError: 275168:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs22954: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 4346.972636] LustreError: 275168:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs22954 [ 4346.977115] LustreError: 275168:0:(mgs_handler.c:517:mgs_target_reg()) fs22954-MDT0001: Error checking target index: rc = -13 [ 4346.984125] LustreError: fs22954-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4346.989665] LustreError: 278578:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 4346.994523] LustreError: 278578:0:(tgt_mount.c:2084:server_put_super()) no obd fs22954-MDT0001 [ 4346.998696] LustreError: 278578:0:(tgt_mount.c:116:server_deregister_mount()) fs22954-MDT0001 not registered [ 4347.017366] Lustre: server umount fs22954-MDT0001 complete [ 4347.019639] LustreError: 278578:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4350.436557] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4350.476158] Lustre: Setting parameter fs22954-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs22954-MDT0001 [ 4350.482526] Lustre: Skipped 2 previous similar messages [ 4350.505576] Lustre: srv-fs22954-MDT0001: No data found on store. Initialize space. [ 4350.508905] Lustre: Skipped 4 previous similar messages [ 4350.551287] Lustre: fs22954-MDT0001: new disk, initializing [ 4350.552581] Lustre: Skipped 3 previous similar messages [ 4350.588672] Lustre: srv-fs22954-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4350.591642] Lustre: 279044:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22954-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4350.595212] LustreError: 279044:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22954-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4352.076857] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4355.169820] Lustre: server umount fs22954-MDT0001 complete [ 4360.150040] loop1: detected capacity change from 0 to 204800000 [ 4362.012760] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4362.114798] Lustre: 280594:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22954-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4362.118070] LustreError: 280594:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22954-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4363.411662] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4365.335894] Lustre: server umount fs22954-MDT0001 complete [ 4371.267403] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.3@tcp (stopping) [ 4371.939388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4371.948367] Lustre: Skipped 3 previous similar messages [ 4375.266847] Lustre: server umount lustre-MDT0000 complete [ 4378.725520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4378.791860] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4380.529125] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4383.843116] loop1: detected capacity change from 0 to 204800000 [ 4384.241473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4386.263051] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4386.443595] Lustre: 283004:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22954-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4386.449559] LustreError: 283004:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22954-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4387.969780] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4390.174941] Lustre: server umount fs22954-MDT0001 complete [ 4399.589975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4399.594382] LustreError: Skipped 1 previous similar message [ 4404.713763] Lustre: server umount lustre-OST0000 complete [ 4409.826636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4412.535768] Lustre: server umount lustre-MDT0000 complete [ 4414.274289] LustreError: 275160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537620 with bad export cookie 11762759312275139397 [ 4414.280943] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4414.282890] LustreError: 275160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4414.447142] Lustre: server umount lustre-MDT0001 complete [ 4419.458344] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4420.941330] Key type lgssc unregistered [ 4421.113610] LNet: 285312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4421.118829] LNetError: 285312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4421.128543] LNet: Removed LNI 192.168.203.103@tcp [ 4421.588150] Key type .llcrypt unregistered [ 4421.590473] Key type ._llcrypt unregistered [ 4430.955506] Key type ._llcrypt registered [ 4430.956587] Key type .llcrypt registered [ 4430.991435] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4431.281063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4431.306067] alg: No test for adler32 (adler32-zlib) [ 4432.155086] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 4432.238502] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4433.831173] Key type lgssc registered [ 4434.239078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4437.253557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4437.259189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4438.420494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4440.051961] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4443.474083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4443.687939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4445.305633] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4447.541646] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4449.146487] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4452.078261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4452.247292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4453.289867] LustreError: 288147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4453.296734] LustreError: 288147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4454.797388] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4457.331427] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4458.470773] LustreError: 288289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.592706] LustreError: 288148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.607806] LustreError: 288148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4468.031195] LustreError: 288147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4473.145252] LustreError: 288707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4473.157110] LustreError: 288707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4475.642992] loop1: detected capacity change from 0 to 204800000 [ 4475.728280] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4475.733016] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4475.738194] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4477.946175] loop1: detected capacity change from 0 to 204800000 [ 4480.145083] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4480.226540] Lustre: fs13190-OST0000: new disk, initializing [ 4480.229600] Lustre: srv-fs13190-OST0000: No data found on store. Initialize space. [ 4480.234524] Lustre: 289960:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4480.275441] Lustre: fs13190-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4482.149584] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4483.384960] LustreError: 288289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4483.390238] LustreError: 288289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4486.222219] Lustre: server umount fs13190-OST0000 complete [ 4490.988284] loop1: detected capacity change from 0 to 204800000 [ 4493.352288] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4493.460471] Lustre: fs13190-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4495.631841] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4499.234551] loop2: detected capacity change from 0 to 204800000 [ 4499.320355] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4499.323961] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4499.328390] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4499.432870] LustreError: 288707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.440948] LustreError: 288707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4501.507572] loop2: detected capacity change from 0 to 204800000 [ 4504.077612] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4504.127522] LustreError: fs13190-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 4504.135761] LustreError: fs13190-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4504.142320] LustreError: 292964:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 4504.146921] LustreError: 292964:0:(tgt_mount.c:2084:server_put_super()) no obd fs13190-OST0001 [ 4504.150620] LustreError: 292964:0:(tgt_mount.c:116:server_deregister_mount()) fs13190-OST0001 not registered [ 4504.166249] Lustre: server umount fs13190-OST0001 complete [ 4504.168334] LustreError: 292964:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4509.214297] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4509.263849] Lustre: fs13190-OST0001: new disk, initializing [ 4509.265659] Lustre: srv-fs13190-OST0001: No data found on store. Initialize space. [ 4509.268350] Lustre: 293515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4509.288723] Lustre: fs13190-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4511.707145] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4516.077044] Lustre: server umount fs13190-OST0000 complete [ 4519.666737] Lustre: server umount fs13190-OST0001 complete [ 4524.304190] loop1: detected capacity change from 0 to 204800000 [ 4526.773266] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4526.872538] Lustre: fs13190-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4529.196267] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4532.228897] loop2: detected capacity change from 0 to 204800000 [ 4534.583484] LustreError: 288707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4534.595765] LustreError: 288707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 4534.780236] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4537.443354] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4541.434789] Lustre: server umount fs13190-OST0000 complete [ 4545.515561] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 4545.604110] Lustre: server umount fs13190-OST0001 complete [ 4550.625956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4550.631174] LustreError: Skipped 1 previous similar message [ 4550.631470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4550.635340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4550.643209] Lustre: Skipped 1 previous similar message [ 4555.746561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4555.749144] Lustre: Skipped 2 previous similar messages [ 4556.637992] Lustre: server umount lustre-OST0000 complete [ 4560.867095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4560.877148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4560.881148] Lustre: Skipped 1 previous similar message [ 4564.343440] Lustre: server umount lustre-MDT0000 complete [ 4566.200333] LustreError: 288170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537772 with bad export cookie 15262208588930038197 [ 4566.205388] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4566.208198] LustreError: 288170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4566.367447] Lustre: server umount lustre-MDT0001 complete [ 4570.772570] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4572.051322] Key type lgssc unregistered [ 4572.190494] LNet: 299367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4572.195505] LNetError: 299367:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4572.206571] LNet: Removed LNI 192.168.203.103@tcp [ 4572.589166] Key type .llcrypt unregistered [ 4572.591752] Key type ._llcrypt unregistered [ 4586.244265] Key type ._llcrypt registered [ 4586.245611] Key type .llcrypt registered [ 4586.287341] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 4594.063785] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4594.479908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4594.485788] alg: No test for adler32 (adler32-zlib) [ 4595.332680] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 4595.418819] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4596.999176] Key type lgssc registered [ 4597.528174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4614.139088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4614.151345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4615.270345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4615.285975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4615.331144] Lustre: lustre-MDT0000: new disk, initializing [ 4615.359304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4615.368237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4616.998601] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4621.968264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.043359] Lustre: 303166:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4622.079288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4622.082364] Lustre: Skipped 1 previous similar message [ 4622.123397] Lustre: lustre-MDT0001: new disk, initializing [ 4622.172644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4622.194180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4622.199268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4623.814437] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4627.646145] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4629.327254] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4632.672070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4632.844462] Lustre: lustre-OST0000: new disk, initializing [ 4632.846174] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4632.848839] Lustre: 304911:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4632.877516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4635.192889] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4638.732541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4638.738996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4638.792821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4638.966200] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4639.818982] Lustre: Mounted lustre-client [ 4640.228598] Lustre: Unmounted lustre-client [ 4643.809624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4643.809737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4643.816345] LustreError: Skipped 1 previous similar message [ 4643.828118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4643.833563] Lustre: Skipped 1 previous similar message [ 4644.838536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4644.842530] Lustre: Skipped 1 previous similar message [ 4645.471479] Lustre: server umount lustre-OST0000 complete [ 4647.904332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4647.908575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.915227] Lustre: Skipped 1 previous similar message [ 4647.917882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4649.953939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4649.954576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4655.073481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4655.077601] Lustre: Skipped 2 previous similar messages [ 4661.215118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4661.290672] Lustre: server umount lustre-MDT0000 complete [ 4662.636412] LustreError: 303160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537868 with bad export cookie 15703819659211755053 [ 4662.640480] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4662.762115] Lustre: server umount lustre-MDT0001 complete [ 4666.093326] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4666.959728] Key type lgssc unregistered [ 4701.919200] LNet: 1 local NIs in recovery (showing 1): 192.168.203.103@tcp [ 4701.921478] LNet: 306939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4701.925388] LNetError: 306939:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4769.509342] LNet: Removed LNI 192.168.203.103@tcp [ 4769.826812] Key type .llcrypt unregistered [ 4769.828819] Key type ._llcrypt unregistered [ 4777.967297] Key type ._llcrypt registered [ 4777.968497] Key type .llcrypt registered [ 4778.005564] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 4785.200184] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4785.524407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4785.590191] alg: No test for adler32 (adler32-zlib) [ 4786.462402] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 4786.546687] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4788.135295] Key type lgssc registered [ 4788.648462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4802.075960] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:53:28 (1780538008) [ 4805.116661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4805.127443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4806.266947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4806.284751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4806.349259] Lustre: lustre-MDT0000: new disk, initializing [ 4806.394444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4806.407818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4808.281842] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4813.736724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4813.808590] Lustre: 310105:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4813.852431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4813.856349] Lustre: Skipped 1 previous similar message [ 4813.910274] Lustre: lustre-MDT0001: new disk, initializing [ 4813.953545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4813.973149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4813.979266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4815.551805] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4819.254258] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4820.954633] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4824.148246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4824.294429] Lustre: lustre-OST0000: new disk, initializing [ 4824.297349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4824.301770] Lustre: 311850:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4824.345852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4826.494361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4826.498475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4826.535216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4826.790971] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4830.871702] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4836.836691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4836.837385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4836.845084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4836.851991] Lustre: Skipped 1 previous similar message [ 4841.612050] Lustre: server umount lustre-OST0000 complete [ 4845.023948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4845.027284] LustreError: Skipped 1 previous similar message [ 4845.028581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4845.032516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4845.034304] Lustre: Skipped 1 previous similar message [ 4847.072177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4847.072657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4852.192057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4852.193702] Lustre: Skipped 2 previous similar messages [ 4857.311109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4857.312158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4857.316138] Lustre: Skipped 1 previous similar message [ 4857.372838] Lustre: server umount lustre-MDT0000 complete [ 4858.550035] LustreError: 310096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538064 with bad export cookie 12831597205953711555 [ 4858.553226] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4858.553454] LustreError: 310096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4858.660534] Lustre: server umount lustre-MDT0001 complete [ 4862.328331] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4863.419913] Key type lgssc unregistered [ 4863.546875] LNet: 313484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4863.549261] LNetError: 313484:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4863.563257] LNet: Removed LNI 192.168.203.103@tcp [ 4863.873797] Key type .llcrypt unregistered [ 4863.875482] Key type ._llcrypt unregistered [ 4874.271177] Key type ._llcrypt registered [ 4874.272267] Key type .llcrypt registered [ 4874.314270] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4874.657218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4874.676657] alg: No test for adler32 (adler32-zlib) [ 4875.533878] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 4875.618069] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4877.199162] Key type lgssc registered [ 4877.595779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4880.231949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4880.238103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4881.400242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4882.755741] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4885.454139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4885.579856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4886.859569] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4888.676702] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4889.829600] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4892.018666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4892.128579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4893.982245] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4895.852561] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4914.837682] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4915.694864] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4921.312868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4921.316273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4921.323158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4921.824612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4921.830695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4921.835229] Lustre: Skipped 1 previous similar message [ 4921.837130] Lustre: Skipped 1 previous similar message [ 4923.714909] Lustre: server umount lustre-MDT0000 complete [ 4925.387193] LustreError: 314847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538131 with bad export cookie 4284724608107232346 [ 4925.390633] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4925.393363] LustreError: 314847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4926.776644] LustreError: 314862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4926.783725] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.3@tcp (stopping) [ 4926.785184] LustreError: 314862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4926.787690] Lustre: Skipped 1 previous similar message [ 4926.945378] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.976876] LustreError: 314861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4930.977491] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4930.984676] LustreError: 314861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4930.993469] Lustre: Skipped 1 previous similar message [ 4931.581211] Lustre: server umount lustre-MDT0001 complete [ 4939.368928] Lustre: server umount lustre-OST0000 complete [ 4942.999335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4944.417684] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4945.385206] Lustre: 318616:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.103@tcp [ 4946.096926] Lustre: server umount MGS complete [ 4948.847037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4948.991646] LustreError: 319186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4949.011364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4950.370234] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4953.180997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4953.310553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4954.621413] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4957.336544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4959.341115] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4961.239505] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4961.852162] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 4962.949331] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4963.464750] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4968.928729] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4968.933018] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4968.938601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4968.940892] Lustre: Skipped 1 previous similar message [ 4969.806317] Lustre: server umount lustre-OST0000 complete [ 4973.137980] Lustre: server umount lustre-MDT0000 complete [ 4974.048947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4974.050074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4974.051139] LustreError: 319183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4974.051151] LustreError: 319183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4974.056316] Lustre: Skipped 2 previous similar messages [ 4974.447042] LustreError: 319167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538180 with bad export cookie 4284724608107233172 [ 4974.450974] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4974.452046] LustreError: 319167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4974.570485] Lustre: server umount lustre-MDT0001 complete [ 4978.215938] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4979.181959] Key type lgssc unregistered [ 4979.300392] LNet: 322361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4979.303067] LNetError: 322361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4979.309302] LNet: Removed LNI 192.168.203.103@tcp [ 4979.612356] Key type .llcrypt unregistered [ 4979.613555] Key type ._llcrypt unregistered [ 4985.404855] Key type ._llcrypt registered [ 4985.406492] Key type .llcrypt registered [ 4985.443571] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 4991.590142] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4992.000110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4992.009261] alg: No test for adler32 (adler32-zlib) [ 4992.866905] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 4992.952667] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4994.527112] Key type lgssc registered [ 4994.874764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5005.845971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5005.854878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5006.947854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5006.959089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5006.994493] Lustre: lustre-MDT0000: new disk, initializing [ 5007.019782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5007.027456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5008.382739] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5012.436147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5012.476653] Lustre: 325462:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5012.499416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5012.502110] Lustre: Skipped 1 previous similar message [ 5012.533501] Lustre: lustre-MDT0001: new disk, initializing [ 5012.561653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5012.572466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5012.577377] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5013.725938] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5016.677629] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5017.763990] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5019.852144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5019.949359] Lustre: lustre-OST0000: new disk, initializing [ 5019.950789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5019.953142] Lustre: 327205:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5019.979190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5020.411890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5020.414698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5020.444088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5021.834317] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5024.873934] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5026.254159] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5026.332107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5027.671363] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5027.743039] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5030.882829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5030.883131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5030.885322] LustreError: Skipped 1 previous similar message [ 5030.890859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5030.894523] Lustre: Skipped 1 previous similar message [ 5034.448145] Lustre: server umount lustre-OST0000 complete [ 5036.000448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.000976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5036.004979] Lustre: Skipped 2 previous similar messages [ 5041.120729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5041.123275] Lustre: Skipped 2 previous similar messages [ 5041.751467] Lustre: server umount lustre-MDT0000 complete [ 5043.038991] LustreError: 325456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538249 with bad export cookie 3726794512625837740 [ 5043.041530] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5043.043087] LustreError: 325456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5043.146556] Lustre: server umount lustre-MDT0001 complete [ 5045.803685] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 21:57:32 (1780538252) [ 5050.596723] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 5053.411263] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5065.129989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5065.221347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5065.231880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5065.267116] Lustre: lustre-MDT0000: new disk, initializing [ 5065.290564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5065.297532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5066.587668] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5070.643157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.681314] Lustre: 331776:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5070.703046] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5070.704865] Lustre: Skipped 1 previous similar message [ 5070.739693] Lustre: lustre-MDT0001: new disk, initializing [ 5070.770197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5070.780868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5070.786061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5072.162466] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5075.185597] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5076.416981] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5078.702271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5078.812871] Lustre: lustre-OST0000: new disk, initializing [ 5078.814401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5078.816715] Lustre: 333520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5078.847144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5079.106892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5079.110037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5079.153352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5080.713768] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5083.731985] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5094.368138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5094.370477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.374577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5098.396227] Lustre: server umount lustre-OST0000 complete [ 5099.679649] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5099.684108] LustreError: Skipped 1 previous similar message [ 5099.686751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5099.693655] Lustre: Skipped 1 previous similar message [ 5099.696795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5099.700027] Lustre: Skipped 1 previous similar message [ 5105.818078] Lustre: server umount lustre-MDT0000 complete [ 5107.000126] LustreError: 334278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538313 with bad export cookie 3726794512625838608 [ 5107.002837] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5107.003274] LustreError: 334278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5107.123065] Lustre: server umount lustre-MDT0001 complete [ 5110.639149] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5111.557878] Key type lgssc unregistered [ 5111.684374] LNet: 335543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5111.686922] LNetError: 335543:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5111.697281] LNet: Removed LNI 192.168.203.103@tcp [ 5111.997339] Key type .llcrypt unregistered [ 5111.998434] Key type ._llcrypt unregistered [ 5142.737620] Key type ._llcrypt registered [ 5142.738695] Key type .llcrypt registered [ 5142.777273] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5142.878207] Key type .llcrypt unregistered [ 5142.879423] Key type ._llcrypt unregistered [ 5174.519543] Key type ._llcrypt registered [ 5174.520693] Key type .llcrypt registered [ 5174.569633] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5174.686392] Key type .llcrypt unregistered [ 5174.687519] Key type ._llcrypt unregistered [ 5189.093311] Key type ._llcrypt registered [ 5189.095142] Key type .llcrypt registered [ 5189.148225] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5189.280863] Key type .llcrypt unregistered [ 5189.282726] Key type ._llcrypt unregistered [ 5202.155935] Key type ._llcrypt registered [ 5202.157008] Key type .llcrypt registered [ 5202.189787] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5202.301214] Key type .llcrypt unregistered [ 5202.302651] Key type ._llcrypt unregistered [ 5235.072052] Key type ._llcrypt registered [ 5235.073341] Key type .llcrypt registered [ 5235.117390] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5235.256270] Key type .llcrypt unregistered [ 5235.258070] Key type ._llcrypt unregistered [ 5247.760117] Key type ._llcrypt registered [ 5247.761351] Key type .llcrypt registered [ 5247.797958] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5247.937486] Key type .llcrypt unregistered [ 5247.939927] Key type ._llcrypt unregistered [ 5261.385609] Key type ._llcrypt registered [ 5261.387586] Key type .llcrypt registered [ 5261.422910] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5261.517209] Key type .llcrypt unregistered [ 5261.519193] Key type ._llcrypt unregistered [ 5273.201890] Key type ._llcrypt registered [ 5273.204775] Key type .llcrypt registered [ 5273.267342] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5273.429295] Key type .llcrypt unregistered [ 5273.431530] Key type ._llcrypt unregistered [ 5286.670476] Key type ._llcrypt registered [ 5286.671796] Key type .llcrypt registered [ 5286.706935] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5286.810418] Key type .llcrypt unregistered [ 5286.812693] Key type ._llcrypt unregistered [ 5298.737464] Key type ._llcrypt registered [ 5298.738547] Key type .llcrypt registered [ 5298.769132] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5298.851260] Key type .llcrypt unregistered [ 5298.852291] Key type ._llcrypt unregistered [ 5325.039835] Key type ._llcrypt registered [ 5325.041949] Key type .llcrypt registered [ 5325.100204] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5325.258364] Key type .llcrypt unregistered [ 5325.260656] Key type ._llcrypt unregistered [ 5339.845820] Key type ._llcrypt registered [ 5339.847042] Key type .llcrypt registered [ 5339.884528] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5339.982451] Key type .llcrypt unregistered [ 5339.984051] Key type ._llcrypt unregistered [ 5354.465244] Key type ._llcrypt registered [ 5354.466629] Key type .llcrypt registered [ 5354.511761] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5354.637298] Key type .llcrypt unregistered [ 5354.638520] Key type ._llcrypt unregistered [ 5379.382756] Key type ._llcrypt registered [ 5379.384217] Key type .llcrypt registered [ 5379.417226] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 5385.918250] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5386.236357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5386.279674] alg: No test for adler32 (adler32-zlib) [ 5387.148758] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 5387.236691] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5388.823174] Key type lgssc registered [ 5389.372940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5404.361600] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5404.372488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5405.514572] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 5405.529260] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 5405.573222] Lustre: tst250fs-MDT0000: new disk, initializing [ 5405.605518] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5405.616514] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5407.126527] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5412.419493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.496571] Lustre: 355223:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5412.546573] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 5412.551502] Lustre: Skipped 1 previous similar message [ 5412.605459] Lustre: tst250fs-MDT0001: new disk, initializing [ 5412.657693] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5412.678253] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5412.685620] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5414.174590] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5416.771182] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5420.093307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5420.271377] Lustre: tst250fs-OST0000: new disk, initializing [ 5420.274250] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 5420.279295] Lustre: 356827:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5420.324744] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5423.113679] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5424.340444] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5424.349848] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5424.406247] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5428.580506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5428.697917] Lustre: tst250fs-OST0001: new disk, initializing [ 5428.701978] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 5428.706278] Lustre: 357684:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5428.765851] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5431.359248] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5431.697077] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5431.700943] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5431.752918] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5434.093529] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 5447.141243] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5447.143363] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5447.148953] Lustre: Skipped 3 previous similar messages [ 5447.152955] Lustre: Skipped 2 previous similar messages [ 5451.445753] Lustre: server umount tst250fs-MDT0000 complete [ 5452.866510] LustreError: 355215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538659 with bad export cookie 2360533992281669187 [ 5452.872639] LustreError: 355215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5452.872771] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5453.034809] Lustre: server umount tst250fs-MDT0001 complete [ 5465.620590] Lustre: server umount tst250fs-OST0000 complete [ 5477.443532] Lustre: server umount tst250fs-OST0001 complete [ 5486.219340] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 5489.838603] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5505.330902] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:05:12 (1780538712) [ 5505.985303] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 5513.639748] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 5516.688461] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5526.252758] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5350 sec ========== 22:05:33 (1780538733) [ 5526.744499] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:05:33 (1780538733) === [ 5528.002989] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:05:34 (1780538734) === [ 5544.954126] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5546.109574] Key type lgssc unregistered [ 5546.227543] LNet: 366698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5546.232405] LNetError: 366698:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5546.241495] LNet: Removed LNI 192.168.203.103@tcp [ 5546.702169] Key type .llcrypt unregistered [ 5546.704630] Key type ._llcrypt unregistered