[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 503603912 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001037] APIC: Switch to symmetric I/O mode setup [ 0.003215] x2apic enabled [ 0.004014] Switched APIC routing to physical x2apic. [ 0.005020] kvm-guest: setup PV IPIs [ 0.008571] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009038] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010029] pid_max: default: 32768 minimum: 301 [ 0.011238] LSM: Security Framework initializing [ 0.012095] Yama: becoming mindful. [ 0.013063] SELinux: Initializing. [ 0.014127] *** VALIDATE selinux *** [ 0.022409] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027083] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029212] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031145] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033170] *** VALIDATE tmpfs *** [ 0.034527] *** VALIDATE proc *** [ 0.035372] *** VALIDATE cgroup *** [ 0.036019] *** VALIDATE cgroup2 *** [ 0.037329] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038187] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041040] Spectre V2 : User space: Vulnerable [ 0.042016] Speculative Store Bypass: Vulnerable [ 0.045265] debug: unmapping init [mem 0xffffffff99259000-0xffffffff99260fff] [ 0.048241] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049877] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050047] ... version: 2 [ 0.051021] ... bit width: 48 [ 0.052017] ... generic registers: 4 [ 0.053020] ... value mask: 0000ffffffffffff [ 0.054022] ... max period: 00007fffffffffff [ 0.055018] ... fixed-purpose events: 3 [ 0.056021] ... event mask: 000000070000000f [ 0.057484] rcu: Hierarchical SRCU implementation. [ 0.060058] smp: Bringing up secondary CPUs ... [ 0.061749] x86: Booting SMP configuration: [ 0.062058] .... node #0, CPUs: #1 #2 #3 [ 0.066309] smp: Brought up 1 node, 4 CPUs [ 0.068027] smpboot: Max logical packages: 1 [ 0.069021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.237099] node 0 deferred pages initialised in 166ms [ 0.241141] devtmpfs: initialized [ 0.242336] x86/mm: Memory block size: 128MB [ 0.246189] gcov: version magic: 0x41383552 [ 0.248315] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.252118] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.255341] pinctrl core: initialized pinctrl subsystem [ 0.257207] [ 0.257681] ************************************************************* [ 0.260017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262012] ** ** [ 0.264015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.266017] ** ** [ 0.268012] ** This means that this kernel is built to expose internal ** [ 0.270012] ** IOMMU data structures, which may compromise security on ** [ 0.272016] ** your system. ** [ 0.275014] ** ** [ 0.277014] ** If you see this message and you are not debugging the ** [ 0.279021] ** kernel, report this immediately to your vendor! ** [ 0.282014] ** ** [ 0.284016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.286014] ************************************************************* [ 0.288970] NET: Registered protocol family 16 [ 0.291489] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.294067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.296072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.300042] cpuidle: using governor menu [ 0.301648] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.304351] PCI: Using configuration type 1 for base access [ 0.307140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.316218] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.318036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.322208] cryptd: max_cpu_qlen set to 1000 [ 0.325306] ACPI: Added _OSI(Module Device) [ 0.326048] ACPI: Added _OSI(Processor Device) [ 0.327178] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.329015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.335209] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.342535] ACPI: Interpreter enabled [ 0.344067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.345013] ACPI: Using IOAPIC for interrupt routing [ 0.347160] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.350451] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.361237] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.363048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.366042] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.371187] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.377875] acpiphp: Slot [2] registered [ 0.379171] acpiphp: Slot [5] registered [ 0.381190] acpiphp: Slot [6] registered [ 0.382116] acpiphp: Slot [7] registered [ 0.383150] acpiphp: Slot [8] registered [ 0.384111] acpiphp: Slot [9] registered [ 0.385074] acpiphp: Slot [10] registered [ 0.386175] acpiphp: Slot [3] registered [ 0.387163] acpiphp: Slot [4] registered [ 0.389192] acpiphp: Slot [11] registered [ 0.390129] acpiphp: Slot [12] registered [ 0.392168] acpiphp: Slot [13] registered [ 0.393110] acpiphp: Slot [14] registered [ 0.394087] acpiphp: Slot [15] registered [ 0.395073] acpiphp: Slot [16] registered [ 0.396135] acpiphp: Slot [17] registered [ 0.398091] acpiphp: Slot [18] registered [ 0.399070] acpiphp: Slot [19] registered [ 0.400014] acpiphp: Slot [20] registered [ 0.401000] acpiphp: Slot [21] registered [ 0.401000] acpiphp: Slot [22] registered [ 0.401000] acpiphp: Slot [23] registered [ 0.403071] acpiphp: Slot [24] registered [ 0.404014] acpiphp: Slot [25] registered [ 0.404933] acpiphp: Slot [26] registered [ 0.406083] acpiphp: Slot [27] registered [ 0.407100] acpiphp: Slot [28] registered [ 0.408070] acpiphp: Slot [29] registered [ 0.408949] acpiphp: Slot [30] registered [ 0.410084] acpiphp: Slot [31] registered [ 0.410941] PCI host bridge to bus 0000:00 [ 0.411012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.413026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.415023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.417030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.419040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.422033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.424203] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.427141] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.431361] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.438021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.442079] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.444018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.445013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.447013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.448666] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.450562] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.454062] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.458861] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.465023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.478019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.483017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.488679] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.495028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.501023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.517030] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.525840] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.531028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.538023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.554019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.562590] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.569024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.575020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.588025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.595422] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.604022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.608016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.621021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.630677] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.636023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.642021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.656022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.664868] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.677024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.689018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.710020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.722084] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.724311] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.726270] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.727256] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.729222] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.733128] iommu: Default domain type: Passthrough [ 0.736307] SCSI subsystem initialized [ 0.737147] ACPI: bus type USB registered [ 0.738111] usbcore: registered new interface driver usbfs [ 0.740062] usbcore: registered new interface driver hub [ 0.741078] usbcore: registered new device driver usb [ 0.742175] pps_core: LinuxPPS API ver. 1 registered [ 0.744010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.746053] PTP clock support registered [ 0.748034] EDAC MC: Ver: 3.0.0 [ 0.749122] PCI: Using ACPI for IRQ routing [ 0.750816] NetLabel: Initializing [ 0.752013] NetLabel: domain hash size = 128 [ 0.753011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.755097] NetLabel: unlabeled traffic allowed by default [ 0.757050] vgaarb: loaded [ 0.757700] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.759016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.765349] clocksource: Switched to clocksource kvm-clock [ 0.900394] VFS: Disk quotas dquot_6.6.0 [ 0.902725] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.905015] *** VALIDATE ramfs *** [ 0.906363] *** VALIDATE hugetlbfs *** [ 0.907766] pnp: PnP ACPI init [ 0.910434] pnp: PnP ACPI: found 6 devices [ 0.928882] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.932075] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.934021] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.935932] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.937623] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.939665] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.941807] NET: Registered protocol family 2 [ 0.943771] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.948150] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.950670] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.955688] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.958690] TCP: Hash tables configured (established 65536 bind 65536) [ 0.961127] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.964092] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.966422] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.969438] NET: Registered protocol family 1 [ 0.971723] RPC: Registered named UNIX socket transport module. [ 0.973722] RPC: Registered udp transport module. [ 0.975275] RPC: Registered tcp transport module. [ 0.976753] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.979750] NET: Registered protocol family 44 [ 0.981298] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.984194] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.986527] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.989653] PCI: CLS 0 bytes, default 64 [ 0.991810] Unpacking initramfs... [ 2.441100] debug: unmapping init [mem 0xffff8a067cc54000-0xffff8a067ffbffff] [ 2.445042] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.447499] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.450485] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.932954] Initialise system trusted keyrings [ 2.935026] Key type blacklist registered [ 2.938439] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.950480] zbud: loaded [ 2.953708] *** VALIDATE nfs *** [ 2.955482] *** VALIDATE nfs4 *** [ 2.957179] pstore: using deflate compression [ 2.960559] Platform Keyring initialized [ 3.064224] NET: Registered protocol family 38 [ 3.065902] Key type asymmetric registered [ 3.068120] Asymmetric key parser 'x509' registered [ 3.069907] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.073154] io scheduler mq-deadline registered [ 3.074769] io scheduler kyber registered [ 3.076264] io scheduler bfq registered [ 3.078235] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.081059] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.084564] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.090773] ACPI: Power Button [PWRF] [ 3.099863] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.107685] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.122335] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.129859] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.148739] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.176475] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.204042] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.210197] Non-volatile memory driver v1.3 [ 3.211784] Linux agpgart interface v0.103 [ 3.250148] virtio_blk virtio1: [vda] 150400 512-byte logical blocks (77.0 MB/73.4 MiB) [ 3.253422] vda: detected capacity change from 0 to 77004800 [ 3.269736] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.273278] vdb: detected capacity change from 0 to 1073741824 [ 3.289753] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.292897] vdc: detected capacity change from 0 to 2621440000 [ 3.309592] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.312793] vdd: detected capacity change from 0 to 2621440000 [ 3.326273] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.328610] vde: detected capacity change from 0 to 4294967296 [ 3.347806] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.350090] vdf: detected capacity change from 0 to 4294967296 [ 3.355841] libphy: Fixed MDIO Bus: probed [ 3.368650] usbcore: registered new interface driver usbserial_generic [ 3.370778] usbserial: USB Serial support registered for generic [ 3.372764] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.377432] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.379077] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.381228] mousedev: PS/2 mouse device common for all mice [ 3.384298] rtc_cmos 00:05: RTC can wake from S4 [ 3.388063] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.388141] rtc_cmos 00:05: registered as rtc0 [ 3.394178] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.395771] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.400297] intel_pstate: CPU model not supported [ 3.406309] hid: raw HID events driver (C) Jiri Kosina [ 3.411835] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.416407] usbcore: registered new interface driver usbhid [ 3.416421] usbhid: USB HID core driver [ 3.416592] drop_monitor: Initializing network drop monitor service [ 3.416760] Initializing XFRM netlink socket [ 3.417231] NET: Registered protocol family 10 [ 3.418574] Segment Routing with IPv6 [ 3.436656] NET: Registered protocol family 17 [ 3.440035] mpls_gso: MPLS GSO support [ 3.455179] RAS: Correctable Errors collector initialized. [ 3.456938] AVX version of gcm_enc/dec engaged. [ 3.458550] AES CTR mode by8 optimization enabled [ 3.547312] sched_clock: Marking stable (3547285594, 0)->(4474674814, -927389220) [ 3.551035] registered taskstats version 1 [ 3.553432] Loading compiled-in X.509 certificates [ 3.555710] zswap: loaded using pool lzo/zbud [ 3.584069] Key type big_key registered [ 3.599334] Key type encrypted registered [ 3.601188] ima: No TPM chip found, activating TPM-bypass! [ 3.603423] ima: Allocated hash algorithm: sha1 [ 3.605352] ima: No architecture policies found [ 3.607371] evm: Initialising EVM extended attributes: [ 3.609628] evm: security.selinux [ 3.611123] evm: security.ima [ 3.612412] evm: security.capability [ 3.613893] evm: HMAC attrs: 0x1 [ 3.618244] rtc_cmos 00:05: setting system clock to 2026-10-01 23:07:52 UTC (1790896072) [ 3.625506] debug: unmapping init [mem 0xffffffff9a203000-0xffffffff9a3fffff] [ 3.628922] debug: unmapping init [mem 0xffffffff98f82000-0xffffffff99258fff] [ 3.636150] Write protecting the kernel read-only data: 28672k [ 3.639535] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 3.642698] debug: unmapping init [mem 0xffffffff97f14000-0xffffffff97ffffff] [ 3.685943] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.694617] systemd[1]: Detected virtualization kvm. [ 3.696870] systemd[1]: Detected architecture x86-64. [ 3.698973] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.728115] systemd[1]: No hostname configured. [ 3.729936] systemd[1]: Set hostname to . [ 3.731975] random: systemd: uninitialized urandom read (16 bytes read) [ 3.734617] systemd[1]: Initializing machine ID from random generator. [ 3.786636] random: ln: uninitialized urandom read (6 bytes read) [ 3.885952] random: systemd: uninitialized urandom read (16 bytes read) [ 3.888503] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.894097] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.898872] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.586694] device-mapper: uevent: version 1.0.3 [ 4.589326] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.384903] virtio_net virtio0 ens2: renamed from eth0 [ 5.387161] random: fast init done [ 5.488634] scsi host0: ata_piix [ 5.508863] scsi host1: ata_piix [ 5.511175] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.513644] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.335156] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.120796] random: crng init done [ 10.122345] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.619504] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.687357] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.896144] SELinux: Disabled at runtime. [ 11.957308] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.966285] systemd[1]: Detected virtualization kvm. [ 11.968270] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.401098] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.403699] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.408412] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.411370] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.413733] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.422487] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.438632] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK [[ 12.523273] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.838270] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.140344] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.162358] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.290277] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.307248] EDAC sbridge: Ver: 1.1.2 [ 14.642271] Key type dns_resolver registered [ 14.948348] NFS: Registering the id_resolver key type [ 14.950619] Key type id_resolver registered [ 14.952422] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg246-server login: [ 51.501319] hrtimer: interrupt took 3297818 ns [ 89.210946] libcfs: loading out-of-tree module taints kernel. [ 89.308876] Key type ._llcrypt registered [ 89.310626] Key type .llcrypt registered [ 89.452195] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 112.742972] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 114.755526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.781341] alg: No test for adler32 (adler32-zlib) [ 116.731305] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 117.744522] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 119.647473] Key type lgssc registered [ 122.012418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 142.376843] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 187.935574] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 202.223376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 202.267839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.563510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 203.633900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 203.825122] Lustre: lustre-MDT0000: new disk, initializing [ 203.973950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.000616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 208.293355] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.718833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.884273] Lustre: 6530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 224.927915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 224.933371] Lustre: Skipped 1 previous similar message [ 225.123145] Lustre: lustre-MDT0001: new disk, initializing [ 225.268146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.358610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 225.362403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 230.898313] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.485609] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 249.428200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.701362] Lustre: lustre-OST0000: new disk, initializing [ 249.705528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 249.711818] Lustre: 8467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.837400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 255.547386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 255.554396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 255.661667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 257.533536] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.192126] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 276.309893] Lustre: lustre-OST0001: new disk, initializing [ 276.317864] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 276.325420] Lustre: 9540:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 276.393785] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 282.176719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 282.189950] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 282.319834] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 285.073797] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 301.247764] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 308.770569] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 318.062903] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing check_logdir /tmp/testlogs/ [ 323.845632] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing yml_node [ 328.609812] Lustre: DEBUG MARKER: Client: 2.17.58.63 [ 331.138814] Lustre: DEBUG MARKER: MDS: 2.17.58.63 [ 334.109434] Lustre: DEBUG MARKER: OSS: 2.17.58.63 [ 335.879333] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Oct 1 19:13:22 EDT 2026 [ 355.806876] Lustre: DEBUG MARKER: only running tests: 121-999 [ 358.056332] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 360.109897] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 378.857696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 378.865585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.886262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 379.873359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 379.873594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.052860] Lustre: server umount lustre-MDT0000 complete [ 387.205556] LustreError: 6521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790896456 with bad export cookie 12103702911587150470 [ 387.206656] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 387.226239] LustreError: 6521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 390.112931] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 390.113317] LustreError: 6537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.114835] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 390.114845] Lustre: Skipped 3 previous similar messages [ 390.138214] Lustre: Skipped 3 previous similar messages [ 390.185806] LustreError: 6537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 392.160866] LustreError: 9548:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 392.164097] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 392.187303] Lustre: Skipped 2 previous similar messages [ 393.841746] Lustre: server umount lustre-MDT0001 complete [ 398.418758] Lustre: server umount lustre-OST0000 complete [ 402.802449] Lustre: server umount lustre-OST0001 complete [ 409.839687] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 419.572973] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 460.637982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.960141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 460.984461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 461.056189] Lustre: lustre-MDT0000: new disk, initializing [ 461.127213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 461.141592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 465.235776] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 476.104623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.276714] Lustre: 15143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 476.355920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 476.362621] Lustre: Skipped 1 previous similar message [ 476.522295] Lustre: lustre-MDT0001: new disk, initializing [ 476.667940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 476.729325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 476.738222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 481.611661] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 491.000801] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 497.105975] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 504.713259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 505.072851] Lustre: lustre-OST0000: new disk, initializing [ 505.084203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 505.094888] Lustre: 17114:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 505.218932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 506.853083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 506.865318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 507.006305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 511.605304] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 522.148831] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 528.345672] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 528.667755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 533.690472] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 533.907201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 537.569691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 537.573701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.585132] LustreError: Skipped 1 previous similar message [ 537.589151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 537.602881] Lustre: Skipped 1 previous similar message [ 547.808970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 547.819585] Lustre: Skipped 4 previous similar messages [ 550.367213] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 550.585306] Lustre: server umount lustre-OST0000 complete [ 558.050129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.054929] Lustre: Skipped 1 previous similar message [ 560.448529] Lustre: server umount lustre-MDT0000 complete [ 563.177622] LustreError: 15156:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.206364] LustreError: 15156:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 563.876555] LustreError: 15137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790896632 with bad export cookie 12103702911587152003 [ 563.877969] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.885915] LustreError: 15137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 564.278397] Lustre: server umount lustre-MDT0001 complete [ 570.164198] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 19:17:16 (1790896636) [ 588.784401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 589.114462] LustreError: 19989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 589.179432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 593.674098] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 594.417509] LustreError: 19990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.969143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.577922] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 616.870474] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 623.439073] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 626.127962] Lustre: Failing over lustre-MDT0000 [ 626.426749] Lustre: server umount lustre-MDT0000 complete [ 628.192689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.197909] LustreError: 19985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.209275] Lustre: Skipped 1 previous similar message [ 628.227944] LustreError: 19985:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 644.575222] Lustre: 3661:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790896697/real 1790896697] req@ffff8a05c4012a00 x1877890759302400/t0(0) o400->MGC192.168.202.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790896713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 644.615436] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.632854] LustreError: 19989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.661862] LustreError: 19989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 647.641447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.841813] LustreError: 3658:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a06c7085500 x1877890759305600/t0(0) o250->MGC192.168.202.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 656.202336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 656.207441] Lustre: Skipped 1 previous similar message [ 656.247743] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 661.475529] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 661.490123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 661.526487] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 661.607413] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 672.673569] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 678.884684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 678.888368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 678.910079] Lustre: Skipped 1 previous similar message [ 678.923632] Lustre: Skipped 3 previous similar messages [ 683.428631] Lustre: server umount lustre-MDT0000 complete [ 684.000359] LustreError: 19990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 684.013661] LustreError: 19990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 687.061748] LustreError: 19971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790896755 with bad export cookie 12103702911587153417 [ 687.063040] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 687.070401] LustreError: 19971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 687.357599] Lustre: server umount lustre-MDT0001 complete [ 696.299793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.712514] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 711.346344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.651723] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 725.176058] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 731.845207] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 733.984451] Lustre: Failing over lustre-MDT0000 [ 734.288221] Lustre: server umount lustre-MDT0000 complete [ 737.056949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.612125] LustreError: 23327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 752.612750] Lustre: 3660:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790896805/real 1790896805] req@ffff8a06fc7bad80 x1877890759333120/t0(0) o400->MGC192.168.202.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790896821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 752.631787] LustreError: 23327:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 752.683557] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 753.380653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 763.877758] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa7f8fdab800379b3 [ 763.897950] Lustre: MGC192.168.202.146@tcp: Connection restored to 0@lo (at 0@lo) [ 763.915226] Lustre: Skipped 1 previous similar message [ 764.426333] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 769.248532] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 769.506584] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 769.510328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 769.555603] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 778.933607] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 784.866949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 784.878361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 790.048708] Lustre: server umount lustre-MDT0000 complete [ 794.231072] LustreError: 23307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790896863 with bad export cookie 12103702911587154355 [ 794.231879] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 794.238562] LustreError: 23307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 794.597812] Lustre: server umount lustre-MDT0001 complete [ 805.096668] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 19:21:11 (1790896871) [ 824.413698] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 836.028468] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 881.952370] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 894.088954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 894.339736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 894.366810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 894.538749] Lustre: lustre-MDT0000: new disk, initializing [ 894.643036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 894.652633] Lustre: Skipped 3 previous similar messages [ 894.675061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 900.031874] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 913.253650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.387867] Lustre: 29705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 913.657085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 919.466956] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 930.398526] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 936.746368] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 943.595279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 943.900966] Lustre: 31675:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 945.170419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 945.203674] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 945.286906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 945.305607] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 949.289696] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 958.012706] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 963.723339] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 986.082886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 986.089276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.098602] Lustre: Skipped 3 previous similar messages [ 986.102598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 986.106914] Lustre: Skipped 2 previous similar messages [ 988.632429] Lustre: server umount lustre-OST0000 complete [ 995.809384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 995.827405] LustreError: Skipped 1 previous similar message [ 998.478411] Lustre: server umount lustre-MDT0000 complete [ 1001.957413] LustreError: 30561:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.975561] LustreError: 30561:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 1002.269027] LustreError: 29696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790897071 with bad export cookie 12103702911587154761 [ 1002.272364] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.284235] LustreError: 29696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1002.710779] Lustre: server umount lustre-MDT0001 complete [ 1015.861478] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 1019.128954] Key type lgssc unregistered [ 1019.574108] LNet: 33651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.587740] LNetError: 33651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.623515] LNet: Removed LNI 192.168.202.146@tcp [ 1020.752161] Key type .llcrypt unregistered [ 1020.753655] Key type ._llcrypt unregistered [ 1048.600695] Key type ._llcrypt registered [ 1048.603110] Key type .llcrypt registered [ 1048.689973] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1049.812914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1049.866180] alg: No test for adler32 (adler32-zlib) [ 1051.224685] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 1051.614413] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1053.464303] Key type lgssc registered [ 1055.151130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1065.087605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1065.127548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.714631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1072.232451] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1076.194717] LustreError: 35083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1076.222889] LustreError: 35083:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1081.320418] LustreError: 35084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.800533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.269399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1086.878802] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1095.740270] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1101.282379] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1108.664372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1108.982093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1112.152167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1115.403347] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1123.515460] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1127.478526] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1141.878720] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 19:26:48 (1790897208) [ 1143.728537] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1145.553531] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1152.899113] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 19:26:59 (1790897219) [ 1160.037864] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 19:27:06 (1790897226) [ 1160.919636] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1169.660565] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1169.664809] Lustre: Skipped 2 previous similar messages [ 1171.254694] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 19:27:18 (1790897238) [ 1174.856305] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1174.860230] Lustre: Skipped 3 previous similar messages [ 1190.894231] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1190.900183] Lustre: Skipped 6 previous similar messages [ 1193.509924] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 19:27:40 (1790897260) [ 1196.194921] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1199.402671] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1207.090515] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 19:27:54 (1790897274) [ 1208.858899] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1208.863411] Lustre: Skipped 3 previous similar messages [ 1221.372738] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 19:28:08 (1790897288) [ 1240.048933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1240.053983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.056537] LustreError: Skipped 1 previous similar message [ 1240.057214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1240.091797] Lustre: Skipped 1 previous similar message [ 1242.447161] Lustre: server umount lustre-OST0000 complete [ 1250.275120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.307808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1250.310554] Lustre: Skipped 2 previous similar messages [ 1252.630193] Lustre: server umount lustre-MDT0000 complete [ 1255.393947] LustreError: 35079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1255.419234] LustreError: 35079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1256.072480] LustreError: 35065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790897324 with bad export cookie 2158092403458057149 [ 1256.086125] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1256.091994] LustreError: 35065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1256.339815] Lustre: server umount lustre-MDT0001 complete [ 1266.946386] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 1269.308231] Key type lgssc unregistered [ 1269.637684] LNet: 42939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.650034] LNetError: 42939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1270.694523] LNet: Removed LNI 192.168.202.146@tcp [ 1271.641961] Key type .llcrypt unregistered [ 1271.649304] Key type ._llcrypt unregistered [ 1289.237839] Key type ._llcrypt registered [ 1289.242933] Key type .llcrypt registered [ 1289.370080] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 1305.548547] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1306.729767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1306.778336] alg: No test for adler32 (adler32-zlib) [ 1307.924892] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 1308.298556] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1310.095304] Key type lgssc registered [ 1311.593587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1347.860454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1347.882851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.111199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1349.134678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1349.208486] Lustre: lustre-MDT0000: new disk, initializing [ 1349.261398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1349.272398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1353.605984] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1365.855087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.016454] Lustre: 46110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1366.110800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1366.116506] Lustre: Skipped 1 previous similar message [ 1366.292771] Lustre: lustre-MDT0001: new disk, initializing [ 1366.500932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1366.574940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1366.596042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1372.560808] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1384.266710] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1391.179218] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1398.627686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1398.793193] Lustre: lustre-OST0000: new disk, initializing [ 1398.795550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1398.799487] Lustre: 48077:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1398.863455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.490620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1404.506871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1404.624515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1406.325884] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1415.961544] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1422.059453] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1422.274276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1427.761671] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1428.166415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1435.616482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1435.619160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.634486] LustreError: Skipped 1 previous similar message [ 1435.667345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1435.680335] Lustre: Skipped 1 previous similar message [ 1435.700937] Lustre: Skipped 1 previous similar message [ 1440.738071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1440.751066] Lustre: Skipped 1 previous similar message [ 1445.343467] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1445.708072] Lustre: server umount lustre-OST0000 complete [ 1450.983470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.994799] Lustre: Skipped 1 previous similar message [ 1451.001908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1451.021091] Lustre: Skipped 1 previous similar message [ 1456.097370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.111425] Lustre: Skipped 1 previous similar message [ 1461.218264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.224295] Lustre: Skipped 1 previous similar message [ 1464.288309] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1466.514531] Lustre: server umount lustre-MDT0000 complete [ 1469.922738] LustreError: 46100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790897538 with bad export cookie 5226725406817537388 [ 1469.926302] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1469.934515] LustreError: 46100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1470.382424] Lustre: server umount lustre-MDT0001 complete [ 1479.796860] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 19:32:27 (1790897547) [ 1486.913682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1487.339378] LustreError: 50540:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.358725] LustreError: 50540:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1487.413688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1491.977757] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1492.450662] LustreError: 50541:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1497.579232] LustreError: 50540:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.705244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1502.121599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1507.207315] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1515.571928] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1521.639655] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1529.535747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1529.804817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.328334] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1545.366977] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1551.920296] LustreError: 53069:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1553.310685] Lustre: Modifying parameter general.timeout=1 in log params [ 1554.859690] Lustre: Modifying parameter general.timeout=2 in log params [ 1557.507946] Lustre: Modifying parameter general.timeout=4 in log params [ 1557.516924] Lustre: Skipped 1 previous similar message [ 1562.259172] Lustre: Modifying parameter general.timeout=8 in log params [ 1562.265492] Lustre: Skipped 3 previous similar messages [ 1570.556337] Lustre: Modifying parameter general.timeout=14 in log params [ 1570.561745] Lustre: Skipped 5 previous similar messages [ 1586.772965] Lustre: Modifying parameter general.timeout=29 in log params [ 1586.784561] Lustre: Skipped 14 previous similar messages [ 1619.437789] Lustre: Modifying parameter general.timeout=64 in log params [ 1619.451247] Lustre: Skipped 34 previous similar messages [ 1684.053197] Lustre: Modifying parameter general.timeout=121 in log params [ 1684.071258] Lustre: Skipped 56 previous similar messages [ 1709.274355] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 19:36:15 (1790897775) [ 1713.152614] Lustre: 60025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1715.224606] Lustre: 60135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1717.494475] Lustre: 60233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1723.665341] Lustre: 60544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1723.683514] Lustre: 60544:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1756.466377] Lustre: 61907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1756.520430] LustreError: 61907:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1761.110817] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 19:37:08 (1790897828) [ 1774.563416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1774.575315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1774.583477] Lustre: Skipped 2 previous similar messages [ 1774.588025] Lustre: Skipped 2 previous similar messages [ 1780.245180] Lustre: server umount lustre-MDT0000 complete [ 1784.201278] LustreError: 53119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790897853 with bad export cookie 5226725406817538270 [ 1784.204369] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1784.211186] LustreError: 53119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1784.468863] Lustre: server umount lustre-MDT0001 complete [ 1799.059441] Lustre: server umount lustre-OST0000 complete [ 1828.172546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.659118] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1828.701167] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1828.711994] Lustre: Skipped 31 previous similar messages [ 1828.913908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.867301] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1844.661817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.897752] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1844.967653] Lustre: 64438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1851.381879] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1861.120657] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1867.113706] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1874.535669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1874.706527] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1874.968324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.974815] Lustre: Skipped 1 previous similar message [ 1881.845095] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1890.845313] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1892.727168] Lustre: 67005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1901.540180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1901.546432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.569695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1901.580226] Lustre: Skipped 3 previous similar messages [ 1905.210863] Lustre: server umount lustre-OST0000 complete [ 1911.777513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1911.784263] LustreError: Skipped 1 previous similar message [ 1911.790055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.809867] Lustre: Skipped 1 previous similar message [ 1916.034508] Lustre: server umount lustre-MDT0000 complete [ 1917.420667] LustreError: 64444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1917.446815] LustreError: 64444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1920.436607] LustreError: 64429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790897989 with bad export cookie 5226725406817541490 [ 1920.443260] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1920.445957] LustreError: 64429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1920.689378] Lustre: server umount lustre-MDT0001 complete [ 1933.422664] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 1936.953630] Key type lgssc unregistered [ 1937.391584] LNet: 68211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.407465] LNetError: 68211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.471475] LNet: Removed LNI 192.168.202.146@tcp [ 1939.890928] Key type .llcrypt unregistered [ 1939.893815] Key type ._llcrypt unregistered [ 1969.418260] Key type ._llcrypt registered [ 1969.423051] Key type .llcrypt registered [ 1969.658425] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1971.958321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1972.223202] alg: No test for adler32 (adler32-zlib) [ 1973.826679] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 1974.399159] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1976.151291] Key type lgssc registered [ 1977.100371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1984.555764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1984.582981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.440595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.923188] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1996.782654] LustreError: 69655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.814153] LustreError: 69655:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2001.680748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2001.891492] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2002.580976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2007.952160] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2015.292477] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2022.477054] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2030.901850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2031.433545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.685786] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2047.040846] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2054.770450] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2063.840225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2063.844181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.862295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2068.456558] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2068.469418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2068.483897] Lustre: Skipped 1 previous similar message [ 2068.497357] Lustre: Skipped 2 previous similar messages [ 2070.232751] Lustre: server umount lustre-MDT0000 complete [ 2073.569390] LustreError: 69652:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.593234] LustreError: 69652:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2074.125838] LustreError: 69637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790898143 with bad export cookie 10237455205537479864 [ 2074.132213] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2074.137616] LustreError: 69637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2074.393921] Lustre: server umount lustre-MDT0001 complete [ 2087.438612] Lustre: server umount lustre-OST0000 complete [ 2117.143419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.562818] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2117.616250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2117.900623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.856280] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2132.858176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.967661] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2133.016949] Lustre: 74529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2133.723429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2138.613629] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2146.176496] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2153.307273] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2162.948557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2163.223285] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2163.689487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2170.504709] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2179.333340] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2181.956533] Lustre: 77105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2184.762490] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2193.377363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2193.382969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.388870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2193.409125] Lustre: Skipped 1 previous similar message [ 2193.423151] Lustre: Skipped 1 previous similar message [ 2197.701475] Lustre: server umount lustre-OST0000 complete [ 2203.617613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.633024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2203.640398] Lustre: Skipped 1 previous similar message [ 2203.658155] Lustre: Skipped 1 previous similar message [ 2208.739781] LustreError: 74541:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.020817] Lustre: server umount lustre-MDT0000 complete [ 2213.324290] LustreError: 76409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790898282 with bad export cookie 10237455205537480704 [ 2213.338589] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2213.343699] LustreError: 76409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2213.663734] Lustre: server umount lustre-MDT0001 complete [ 2229.144407] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2232.799314] Key type lgssc unregistered [ 2233.162390] LNet: 78319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2233.176269] LNetError: 78319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.243822] LNet: Removed LNI 192.168.202.146@tcp [ 2235.700951] Key type .llcrypt unregistered [ 2235.702958] Key type ._llcrypt unregistered [ 2263.870240] Key type ._llcrypt registered [ 2263.873327] Key type .llcrypt registered [ 2263.960810] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2265.143214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2265.375171] alg: No test for adler32 (adler32-zlib) [ 2266.356435] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 2266.574665] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2268.191296] Key type lgssc registered [ 2269.262398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2276.598719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2276.614168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.209836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2283.294501] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2287.588911] LustreError: 79747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.625016] LustreError: 79747:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2291.707189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2292.395975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2296.698569] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2305.032855] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2311.916767] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2318.772245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2319.317144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2326.928444] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2334.994708] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2341.228906] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2342.066750] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2343.724673] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2343.730654] Lustre: Skipped 1 previous similar message [ 2346.287700] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2346.290437] Lustre: Skipped 2 previous similar messages [ 2351.008421] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2351.016446] Lustre: Skipped 4 previous similar messages [ 2359.630684] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2359.639330] Lustre: Skipped 7 previous similar messages [ 2375.811547] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2375.821847] Lustre: Skipped 16 previous similar messages [ 2407.814454] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2407.818383] Lustre: Skipped 35 previous similar messages [ 2442.990887] Lustre: 87849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2445.714588] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:48:32 (1790898512) [ 2447.098628] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2448.846506] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:48:35 (1790898515) [ 2454.498456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2454.504396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2454.512842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2454.523722] Lustre: Skipped 1 previous similar message [ 2459.635795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2459.644434] Lustre: Skipped 2 previous similar messages [ 2464.755639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2464.770320] Lustre: Skipped 1 previous similar message [ 2467.295218] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2467.550657] Lustre: server umount lustre-OST0000 complete [ 2471.393401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2471.404940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.422772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2474.984336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.297302] Lustre: server umount lustre-MDT0000 complete [ 2480.097135] LustreError: 79747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.970588] LustreError: 79729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790898551 with bad export cookie 3939397390592454623 [ 2482.973211] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2482.983571] LustreError: 79729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2483.336486] Lustre: server umount lustre-MDT0001 complete [ 2494.979535] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2497.687966] Key type lgssc unregistered [ 2498.048890] LNet: 89098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2498.064574] LNetError: 89098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2498.098291] LNet: Removed LNI 192.168.202.146@tcp [ 2498.954616] Key type .llcrypt unregistered [ 2498.956458] Key type ._llcrypt unregistered [ 2503.704952] Key type ._llcrypt registered [ 2503.710382] Key type .llcrypt registered [ 2503.808935] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2508.390737] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules [ 2509.459533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2509.687699] alg: No test for adler32 (adler32-zlib) [ 2510.809713] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 2510.996372] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2512.712978] Key type lgssc registered [ 2513.698904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2515.076453] LustreError: 90316:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2532.440975] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2537.176599] LustreError: 90316:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2543.357187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2543.401685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.867607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.210386] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2554.337865] LustreError: 91342:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.356569] LustreError: 91342:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2558.006633] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:50:24 (1790898624) [ 2559.473311] LustreError: 91343:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.931907] LustreError: 91374:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2563.951829] LustreError: 91374:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2563.960141] LustreError: 91374:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2564.340766] Lustre: server umount lustre-MDT0000 complete [ 2581.918582] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2585.942457] Key type lgssc unregistered [ 2586.472715] LNet: 92867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2586.482564] LNetError: 92867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2586.507554] LNet: Removed LNI 192.168.202.146@tcp [ 2587.733198] Key type .llcrypt unregistered [ 2587.737527] Key type ._llcrypt unregistered [ 2610.687287] Key type ._llcrypt registered [ 2610.694277] Key type .llcrypt registered [ 2610.981325] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2612.503531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2612.521210] alg: No test for adler32 (adler32-zlib) [ 2613.544089] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 2613.908881] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2615.640132] Key type lgssc registered [ 2617.585977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2626.261384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2626.286756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.846835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2633.217793] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2637.283537] LustreError: 94106:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.302089] LustreError: 94106:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2642.404991] LustreError: 94105:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.430301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.179125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2649.608253] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2658.826763] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2666.458455] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2675.001073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2675.739400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.481760] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2690.094182] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2753.068678] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:53:40 (1790898820) [ 2755.178691] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2757.047599] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:53:44 (1790898824) [ 2761.698990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2761.708232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2761.732538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2764.776385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2764.789651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2764.804031] Lustre: Skipped 1 previous similar message [ 2764.814380] Lustre: Skipped 1 previous similar message [ 2767.600372] Lustre: server umount lustre-MDT0000 complete [ 2769.898116] LustreError: 94105:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.917129] LustreError: 94105:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2771.478785] LustreError: 94087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790898840 with bad export cookie 1889128494790127012 [ 2771.497140] LustreError: 94087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2771.501613] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.871232] Lustre: server umount lustre-MDT0001 complete [ 2787.087052] Lustre: server umount lustre-OST0000 complete [ 2802.557205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.247124] LustreError: 98424:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2803.467541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2808.593921] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2808.804571] LustreError: 98425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.422069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.520751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2825.215819] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2835.176385] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2842.753723] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2858.983913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2859.319828] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2859.332360] LustreError: 98414:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2859.370321] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2859.389246] LustreError: 100508:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2859.395041] LustreError: 100508:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2859.403530] LustreError: 100508:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2859.518613] Lustre: server umount lustre-OST0000 complete [ 2859.523210] LustreError: 100508:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2868.301267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2868.517225] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2868.522927] LustreError: 98413:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2868.576096] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2868.584318] LustreError: 100975:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2868.594390] LustreError: 100975:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2868.599735] LustreError: 100975:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2868.678324] Lustre: server umount lustre-OST0000 complete [ 2868.680391] LustreError: 100975:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2878.643841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2878.846709] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2878.852506] Lustre: Found index 0 for lustre-OST0000, updating log [ 2878.859232] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2878.984582] Lustre: lustre-OST0000: new disk, initializing [ 2878.988456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2879.003312] Lustre: 101511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2879.366346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2887.030112] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2891.590195] Lustre: Failing over lustre-OST0000 [ 2891.778931] Lustre: server umount lustre-OST0000 complete [ 2895.841752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2895.852149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.862350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2899.940954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2899.951854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2902.493636] Lustre: server umount lustre-MDT0000 complete [ 2905.064028] LustreError: 99155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.089802] LustreError: 99155:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2906.963081] LustreError: 100527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790898975 with bad export cookie 1889128494790129756 [ 2906.971778] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2906.977316] LustreError: 100527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2916.575779] Lustre: server umount lustre-MDT0001 complete [ 2926.481701] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:56:33 (1790898993) [ 2946.342604] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2955.336423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.624171] LustreError: 103860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2955.718856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.837569] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2967.987951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.332723] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2976.182846] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2982.734885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2983.338698] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2989.721978] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2993.578588] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2993.627883] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2993.629718] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2998.464271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2998.514965] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2998.576443] Lustre: lustre-OST0001: new disk, initializing [ 2998.578641] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2998.586784] Lustre: 106176:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2998.912115] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2998.914497] Lustre: Skipped 2 previous similar messages [ 3005.619846] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3007.659860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3007.672692] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3007.847182] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3014.714379] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3023.211447] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3028.451919] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.457958] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3028.462185] Lustre: Skipped 1 previous similar message [ 3028.469612] Lustre: Skipped 3 previous similar messages [ 3034.082935] LustreError: 103857:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3034.112668] LustreError: 103857:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3034.275964] Lustre: server umount lustre-MDT0001 complete [ 3044.146948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3044.239927] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3044.256273] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3044.294937] Lustre: 103847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3046.263793] LustreError: 107764:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3046.269832] LustreError: 107764:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3046.281054] LustreError: 103852:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3046.391734] LustreError: 107773:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.146@tcp: cfg command failed: rc = -17 [ 3046.401923] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3046.401923] [ 3049.003655] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3049.988407] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3060.762227] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 19:58:47 (1790899127) [ 3077.773756] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3101.016543] Lustre: 109792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3126.826147] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3130.987790] Lustre: 110927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3234.272729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3234.275888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3234.297786] Lustre: Skipped 2 previous similar messages [ 3234.312588] Lustre: Skipped 4 previous similar messages [ 3236.436776] Lustre: server umount lustre-MDT0000 complete [ 3239.395855] LustreError: 103855:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3239.431793] LustreError: 103855:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 3240.173639] LustreError: 103841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790899309 with bad export cookie 1889128494790130757 [ 3240.175934] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3240.195774] LustreError: 103841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3240.605570] Lustre: server umount lustre-MDT0001 complete [ 3256.124645] Lustre: server umount lustre-OST0000 complete [ 3271.098807] Lustre: server umount lustre-OST0001 complete [ 3278.625606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3291.072907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3302.978906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3313.556902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3338.853246] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3350.762673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.789804] Lustre: lustre-MDT0000: reset Object Index mappings [ 3351.556023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.558441] Lustre: Skipped 1 previous similar message [ 3356.267453] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3365.709072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3365.727451] Lustre: lustre-MDT0001: reset Object Index mappings [ 3370.983842] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3374.679093] Lustre: 116068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3382.494576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3390.607920] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3392.742800] LustreError: 116422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.755944] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 3392.764189] LustreError: 116422:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3397.899765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 3400.853740] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3406.852344] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3406.862725] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3408.890434] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3417.840446] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3422.014588] Lustre: 117945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3475.917869] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 20:05:43 (1790899543) [ 3504.105643] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3504.109887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3504.137058] Lustre: Skipped 4 previous similar messages [ 3504.161283] Lustre: Skipped 3 previous similar messages [ 3508.500857] Lustre: server umount lustre-MDT0000 complete [ 3512.879694] LustreError: 114901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790899581 with bad export cookie 1889128494790177643 [ 3512.886541] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3512.893093] LustreError: 114901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3519.876438] Lustre: server umount lustre-MDT0001 complete [ 3529.866852] Lustre: server umount lustre-OST0000 complete [ 3534.984615] LustreError: 118907:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0x30:0x0].0x0 (ffff8a06ff95c600) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3541.540584] Lustre: server umount lustre-OST0001 complete [ 3549.587182] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 3562.516890] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3601.552563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3601.770206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3601.798776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3601.878094] Lustre: lustre-MDT0000: new disk, initializing [ 3601.931136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3601.942514] Lustre: Skipped 3 previous similar messages [ 3601.957382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3606.508763] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3613.343761] Lustre: server umount lustre-MDT0000 complete [ 3623.675941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.989276] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3624.002212] Lustre: 122219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3624.019757] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3629.569413] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3635.458639] Lustre: server umount lustre-MDT0000 complete [ 3644.303879] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 20:08:31 (1790899711) [ 3645.931539] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3647.876217] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 20:08:34 (1790899714) [ 3666.630958] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3677.354734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3683.473035] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3693.328955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3693.421129] Lustre: 124269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3693.518533] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3693.527379] Lustre: Skipped 1 previous similar message [ 3693.733819] Lustre: lustre-MDT0001: new disk, initializing [ 3693.984359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3694.004159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3699.017383] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3704.155395] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3711.222179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3711.541783] Lustre: lustre-OST0000: new disk, initializing [ 3711.545906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3711.551976] Lustre: 125847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3712.941273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3712.952910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3713.126634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3717.843219] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3730.436113] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3730.775603] Lustre: lustre-OST0001: new disk, initializing [ 3730.780589] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3730.785880] Lustre: 126718:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3731.980582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3731.993063] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3732.144954] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3740.134350] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3751.654712] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3756.301770] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4181.723548] Lustre: 133137:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4181.744445] Lustre: 133137:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4181.756057] Lustre: 133137:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4181.771881] Lustre: 133137:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4181.779098] Lustre: 133137:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4181.788598] Lustre: 133137:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4449.714735] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:21:56 (1790900516) [ 4467.170410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4467.178513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4467.184535] Lustre: Skipped 3 previous similar messages [ 4467.190604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4467.199037] Lustre: Skipped 4 previous similar messages [ 4471.408647] Lustre: server umount lustre-MDT0000 complete [ 4474.868858] LustreError: 124275:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4474.908214] LustreError: 124275:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 4475.491788] LustreError: 133150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790900544 with bad export cookie 1889128494790245788 [ 4475.493551] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4475.502272] LustreError: 133150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4479.984890] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4479.997376] Lustre: Skipped 4 previous similar messages [ 4480.004360] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4480.012513] Lustre: Skipped 4 previous similar messages [ 4482.285831] Lustre: server umount lustre-MDT0001 complete [ 4493.907119] Lustre: server umount lustre-OST0000 complete [ 4512.223143] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4512.460800] Lustre: server umount lustre-OST0001 complete [ 4519.381986] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 4529.328816] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4567.529941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.715497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4567.744508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4567.864592] Lustre: lustre-MDT0000: new disk, initializing [ 4567.992245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.996143] Lustre: Skipped 5 previous similar messages [ 4568.021341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4573.408900] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4583.994924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.098078] Lustre: 136188:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4584.184172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4584.187840] Lustre: Skipped 1 previous similar message [ 4584.244246] Lustre: lustre-MDT0001: new disk, initializing [ 4584.406890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4584.420604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4588.185773] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4596.568381] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4602.352548] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4609.399830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4609.618872] Lustre: lustre-OST0000: new disk, initializing [ 4609.622585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4609.628179] Lustre: 138156:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4609.709981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4609.714960] Lustre: Skipped 1 previous similar message [ 4610.811378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4610.833177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4610.969124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4615.469471] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4625.269526] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4635.758898] Lustre: lustre-MDD0000: changelog on [ 4638.957422] Lustre: lustre-MDD0001: changelog on [ 6703.537924] Lustre: lustre-MDD0001: changelog off [ 6705.399227] Lustre: 136194:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6706.287476] Lustre: lustre-MDD0000: changelog off [ 6713.336668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6713.350114] Lustre: Skipped 1 previous similar message [ 6713.350512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6713.358445] Lustre: Skipped 4 previous similar messages [ 6718.453865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6718.461535] Lustre: Skipped 1 previous similar message [ 6723.552435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6723.563071] Lustre: Skipped 1 previous similar message [ 6726.117751] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6726.315131] Lustre: server umount lustre-OST0000 complete [ 6733.794341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6733.797389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6733.813732] Lustre: Skipped 1 previous similar message [ 6733.831569] Lustre: Skipped 1 previous similar message [ 6737.287572] Lustre: server umount lustre-MDT0000 complete [ 6738.914792] LustreError: 136198:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6738.925060] LustreError: 136198:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6741.103701] LustreError: 136180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790902809 with bad export cookie 1889128494791021619 [ 6741.114775] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6741.126292] LustreError: 136180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6741.490522] Lustre: server umount lustre-MDT0001 complete [ 6753.240324] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 6755.974799] Key type lgssc unregistered [ 6756.245894] LNet: 147330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6756.251739] LNetError: 147330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6756.268732] LNet: Removed LNI 192.168.202.146@tcp [ 6756.914165] Key type .llcrypt unregistered [ 6756.917869] Key type ._llcrypt unregistered [ 6776.430919] Key type ._llcrypt registered [ 6776.432961] Key type .llcrypt registered [ 6776.498973] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 6792.910861] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 6794.395019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6794.469489] alg: No test for adler32 (adler32-zlib) [ 6795.492844] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 6795.754075] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 6797.431376] Key type lgssc registered [ 6798.530535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6835.518949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6835.561146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.807728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6836.846021] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6836.939906] Lustre: lustre-MDT0000: new disk, initializing [ 6837.000079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6837.025442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6841.489640] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6852.026529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6852.129303] Lustre: 150557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6852.181951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6852.186214] Lustre: Skipped 1 previous similar message [ 6852.244701] Lustre: lustre-MDT0001: new disk, initializing [ 6852.321528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6852.353971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6852.367585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6857.021884] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6866.865671] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6873.083290] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6879.689351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6880.036356] Lustre: lustre-OST0000: new disk, initializing [ 6880.039956] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6880.045217] Lustre: 152527:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6880.113152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6885.909381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6885.915755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6886.032233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6887.024367] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6896.489369] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6902.275268] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_module obdecho/obdecho [ 6904.160956] LustreError: 153530:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6904.167506] LustreError: 153530:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6905.739260] LustreError: 153627:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6917.099155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6917.109460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6917.131112] Lustre: Skipped 2 previous similar messages [ 6917.141225] Lustre: Skipped 1 previous similar message [ 6922.208784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6922.214744] Lustre: Skipped 3 previous similar messages [ 6922.636959] Lustre: server umount lustre-MDT0000 complete [ 6926.430676] LustreError: 152525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790902995 with bad export cookie 14237719835450850290 [ 6926.435994] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6926.442638] LustreError: 152525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6926.840921] Lustre: server umount lustre-MDT0001 complete [ 6942.322169] Lustre: server umount lustre-OST0000 complete [ 6966.932908] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 6979.552434] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7021.792070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7022.171638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7022.195732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7022.325867] Lustre: lustre-MDT0000: new disk, initializing [ 7022.469714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7022.509352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7027.866571] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7039.323428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7039.511111] Lustre: 157344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7039.583134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7039.590759] Lustre: Skipped 1 previous similar message [ 7039.713347] Lustre: lustre-MDT0001: new disk, initializing [ 7039.825846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7039.890529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7039.906452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7044.126060] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7054.545816] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7060.903694] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7067.179343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7067.438571] Lustre: lustre-OST0000: new disk, initializing [ 7067.449664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7067.456754] Lustre: 159313:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7067.571464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7073.369537] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7075.418434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7075.426739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7075.510917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7082.955845] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7087.874407] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7088.090540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7092.835787] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7093.071791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7096.288627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7096.290904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7096.295583] LustreError: Skipped 1 previous similar message [ 7096.305174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7096.329716] Lustre: Skipped 1 previous similar message [ 7096.363938] Lustre: Skipped 1 previous similar message [ 7101.166495] Lustre: server umount lustre-OST0000 complete [ 7106.528636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7106.530023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7106.541718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7106.548466] Lustre: Skipped 1 previous similar message [ 7106.558869] Lustre: Skipped 1 previous similar message [ 7110.953440] Lustre: server umount lustre-MDT0000 complete [ 7114.852535] LustreError: 157338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790903183 with bad export cookie 14237719835450851249 [ 7114.853237] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7114.859369] LustreError: 157338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7115.085675] Lustre: server umount lustre-MDT0001 complete [ 7118.733974] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:06:25 (1790903185) [ 7120.609106] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7122.404717] Lustre: DEBUG MARKER: == conf-sanity test 138: MDT mount failure must not LBUG destroying recovery exports ========================================================== 21:06:29 (1790903189) [ 7141.858550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7142.367223] LustreError: 162287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7142.396804] LustreError: 162287:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7142.429827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7146.784847] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7147.491088] LustreError: 162288:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7152.610377] LustreError: 162287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7156.119828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7161.164834] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7169.204972] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7175.667393] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7183.840401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7184.334964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7184.339934] Lustre: Skipped 1 previous similar message [ 7192.368732] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7200.872106] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7209.949908] Lustre: *** cfs_fail_loc=718, val=0*** [ 7210.321129] Lustre: server umount lustre-MDT0000 complete [ 7212.001948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7212.021683] LustreError: 164128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7212.035782] Lustre: Skipped 2 previous similar messages [ 7212.065344] LustreError: 164128:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7213.692509] Lustre: *** cfs_fail_loc=718, val=0*** [ 7213.696479] Lustre: Skipped 3 previous similar messages [ 7213.697597] LustreError: 163021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790903282 with bad export cookie 14237719835450852131 [ 7213.699883] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7213.727503] LustreError: 163021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7214.044637] Lustre: server umount lustre-MDT0001 complete [ 7223.215440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7233.504290] Lustre: 148772:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790903286/real 1790903286] req@00000000d6d8261b x1877897761631360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790903302 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7233.539897] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.527857] Lustre: 148773:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790903287/real 1790903287] req@000000003bee2a64 x1877897761631488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790903303 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7238.623359] Lustre: 148771:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790903291/real 1790903291] req@00000000cba2c2eb x1877897761631616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790903307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7238.697273] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc5968ac17394c648 [ 7238.702043] Lustre: MGC192.168.202.146@tcp: Connection restored to 0@lo (at 0@lo) [ 7238.897771] Lustre: *** cfs_fail_loc=723, val=0*** [ 7238.902875] Lustre: Skipped 1 previous similar message [ 7238.983961] LustreError: 165646:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-122) [ 7238.991963] LustreError: 165646:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.146@tcp: cfg command failed: rc = -122 [ 7238.996061] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7238.996061] [ 7239.002236] LustreError: MGC192.168.202.146@tcp: Confguration from log lustre-MDT0000 failed from MGS -122. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7239.011976] LustreError: 165615:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -122 [ 7239.018286] LustreError: 165615:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -122 [ 7239.023639] LustreError: 165615:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 7239.030225] LustreError: 165615:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 7239.110250] Lustre: server umount lustre-MDT0000 complete [ 7239.114355] LustreError: 165615:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -122 [ 7244.768144] Lustre: 148772:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790903297/real 1790903297] req@000000001513ebc5 x1877897761632000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790903313 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7244.797637] Lustre: 148772:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7245.901495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7246.075644] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7246.327688] LustreError: 164118:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7250.833215] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7252.523937] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7257.582115] LustreError: 164119:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7257.598079] LustreError: 164119:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7258.775308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7259.097309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7259.104749] Lustre: Skipped 1 previous similar message [ 7259.156861] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7263.035827] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7263.190759] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7264.236180] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7264.255583] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7270.039470] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7271.531834] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7278.129154] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7279.886572] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7292.384765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7292.397374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7292.410534] LustreError: 164280:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7292.425385] LustreError: 164280:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7292.479421] Lustre: server umount lustre-OST0000 complete [ 7300.073323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7300.073971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7300.080019] Lustre: Skipped 1 previous similar message [ 7301.853758] Lustre: server umount lustre-MDT0000 complete [ 7304.955269] LustreError: 163021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790903373 with bad export cookie 14237719835450853076 [ 7304.965273] LustreError: 163021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7304.965925] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7314.478785] Lustre: server umount lustre-MDT0001 complete [ 7328.390624] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 7332.167311] Key type lgssc unregistered [ 7332.513585] LNet: 168875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7332.525702] LNetError: 168875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7332.558057] LNet: Removed LNI 192.168.202.146@tcp [ 7333.525411] Key type .llcrypt unregistered [ 7333.526923] Key type ._llcrypt unregistered [ 7353.119355] Key type ._llcrypt registered [ 7353.121898] Key type .llcrypt registered [ 7353.209505] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7353.991112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7354.064055] alg: No test for adler32 (adler32-zlib) [ 7355.224876] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 7355.453257] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 7357.127208] Key type lgssc registered [ 7358.175750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7365.398834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7365.425633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7366.873955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7371.458678] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7376.365031] LustreError: 170206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7376.395931] LustreError: 170206:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7379.841323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.154377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7384.022415] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7391.183719] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7397.061754] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7403.457929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7403.790653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7410.328455] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7412.032619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7417.364267] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7425.519311] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 7426.528839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7426.533042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7426.552300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7427.553894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7427.556990] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7427.564150] Lustre: Skipped 1 previous similar message [ 7427.581735] Lustre: Skipped 1 previous similar message [ 7430.611809] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 7431.516549] Lustre: server umount lustre-MDT0000 complete [ 7432.676226] LustreError: 170201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7432.690533] LustreError: 170201:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7435.171357] LustreError: 170188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790903504 with bad export cookie 3972016395809209959 [ 7435.176466] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7435.181669] LustreError: 170188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7435.755486] LustreError: 172046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7435.770254] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.46@tcp (stopping) [ 7437.795644] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7437.800417] LustreError: 170201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7441.663666] Lustre: server umount lustre-MDT0001 complete [ 7447.757965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7459.522137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7459.786930] LustreError: 174048:0:(llog.c:1655:llog_backup()) MGC192.168.202.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7459.797486] Lustre: 174048:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7465.445628] LustreError: 174052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7465.476658] LustreError: 174052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7475.671773] LustreError: 174054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7475.692609] LustreError: 174054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 7492.051266] LustreError: 174052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7492.067649] LustreError: 174052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 7521.247472] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7521.258832] Lustre: 174048:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7526.869794] LustreError: 174052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7526.899919] LustreError: 174052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 21 previous similar messages [ 7562.233065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7567.342911] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7568.084947] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7577.143529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7580.645231] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x371f6fde0a3c5876 [ 7580.682909] Lustre: MGC192.168.202.146@tcp: Connection restored to 0@lo (at 0@lo) [ 7580.942174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7581.012324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 7585.877697] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7586.284515] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7595.437561] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7597.627291] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7604.311279] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7606.047869] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7627.237509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7627.239245] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7627.249781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7627.250361] Lustre: Skipped 3 previous similar messages [ 7627.265339] Lustre: Skipped 1 previous similar message [ 7628.215887] Lustre: server umount lustre-MDT0000 complete [ 7631.901601] LustreError: 172988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790903700 with bad export cookie 3972016395809216630 [ 7631.906916] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7631.914497] LustreError: 172988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7632.352258] LustreError: 174052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7632.354276] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7632.370812] LustreError: 174052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 39 previous similar messages [ 7632.380814] Lustre: Skipped 1 previous similar message [ 7632.558462] Lustre: server umount lustre-MDT0001 complete [ 7638.633473] Lustre: server umount lustre-OST0000 complete [ 7661.906923] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 7674.574652] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7719.153736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7719.522981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7719.590220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7719.774740] Lustre: lustre-MDT0000: new disk, initializing [ 7719.906310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7719.929299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7725.539753] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7738.793111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7738.977525] Lustre: 179155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7739.107399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7739.112227] Lustre: Skipped 1 previous similar message [ 7739.288583] Lustre: lustre-MDT0001: new disk, initializing [ 7739.475564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7739.524306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7739.537234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7744.550594] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7754.200301] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7760.004115] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7767.806051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7768.120890] Lustre: lustre-OST0000: new disk, initializing [ 7768.124307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7768.132146] Lustre: 181124:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7774.453595] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7777.383068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7777.396503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7777.484072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7784.673544] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7789.782990] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7789.961329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7795.141791] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7795.387401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7798.243661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7798.252754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7798.253897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7798.253913] Lustre: Skipped 2 previous similar messages [ 7798.276157] Lustre: Skipped 1 previous similar message [ 7803.645254] Lustre: server umount lustre-OST0000 complete [ 7822.303146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7822.631686] Lustre: server umount lustre-MDT0000 complete [ 7823.840414] LustreError: 179166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7823.861734] LustreError: 179166:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7826.179675] LustreError: 179147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790903895 with bad export cookie 3972016395809233696 [ 7826.182661] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7826.190469] LustreError: 179147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7826.483667] Lustre: server umount lustre-MDT0001 complete [ 7836.215849] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:18:22 (1790903902) [ 7841.631433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7842.086613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7842.101555] Lustre: Skipped 1 previous similar message [ 7846.196602] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7856.362860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7856.615569] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7856.620861] Lustre: Skipped 9 previous similar messages [ 7862.842731] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7871.257089] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7877.330871] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7883.902082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7891.501535] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7899.423829] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7913.713071] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:19:40 (1790903980) [ 7922.159349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7922.159830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7922.172973] LustreError: Skipped 2 previous similar messages [ 7922.189892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7922.209389] Lustre: Skipped 3 previous similar messages [ 7922.254556] Lustre: Skipped 1 previous similar message [ 7925.238469] Lustre: server umount lustre-OST0000 complete [ 7936.973644] Lustre: server umount lustre-MDT0000 complete [ 7940.736760] LustreError: 183518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790904009 with bad export cookie 3972016395809234578 [ 7940.747870] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7941.031911] Lustre: server umount lustre-MDT0001 complete [ 7953.613254] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 7956.677906] Key type lgssc unregistered [ 7957.067671] LNet: 187267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7957.075663] LNetError: 187267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7957.090941] LNet: Removed LNI 192.168.202.146@tcp [ 7957.965521] Key type .llcrypt unregistered [ 7957.967920] Key type ._llcrypt unregistered [ 7977.735996] Key type ._llcrypt registered [ 7977.738145] Key type .llcrypt registered [ 7977.820445] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7978.903920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7979.134337] alg: No test for adler32 (adler32-zlib) [ 7980.179300] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 7980.527784] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 7982.271731] Key type lgssc registered [ 7983.329713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7990.829234] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7990.843990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8006.367412] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8011.489030] LustreError: 188532:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.146@tcp: failed processing log, type 4: rc = -110 [ 8016.673323] LustreError: 188532:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8016.680921] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8016.686508] LustreError: MGC192.168.202.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8016.696968] LustreError: 188532:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8016.705450] LustreError: 188532:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 8016.720415] LustreError: 188532:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8016.789929] Lustre: server umount lustre-OST0000 complete [ 8016.793820] LustreError: 188532:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8026.248359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8027.741551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8032.483422] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8037.859897] LustreError: 189031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8037.883328] LustreError: 189031:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8039.679362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8039.966397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8044.776528] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8053.012653] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8058.929388] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8065.601191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8065.810241] LustreError: 190846:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8065.816162] Lustre: 190846:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8065.927133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8072.396235] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8080.236711] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8083.429283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8083.438892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8083.442684] Lustre: Skipped 1 previous similar message [ 8083.450667] Lustre: Skipped 1 previous similar message [ 8084.516856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8084.518815] Lustre: Skipped 1 previous similar message [ 8089.569475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8089.588526] Lustre: Skipped 2 previous similar messages [ 8094.693400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8094.712273] Lustre: Skipped 2 previous similar messages [ 8096.223160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8096.803140] Lustre: server umount lustre-MDT0000 complete [ 8099.819319] LustreError: 189026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8099.878662] LustreError: 189026:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8103.865935] LustreError: 190874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790904172 with bad export cookie 1288404474745392087 [ 8103.895470] LustreError: 190874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8103.909494] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8104.928629] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8104.933706] LustreError: 190869:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8104.946723] Lustre: Skipped 1 previous similar message [ 8104.948162] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8105.055618] LustreError: 189760:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a598f434 x1877899004116352/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8105.228982] Lustre: server umount lustre-MDT0001 complete [ 8111.463531] Lustre: server umount lustre-OST0000 complete [ 8124.146762] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 8134.918458] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8172.917277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8173.231166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8173.336284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8173.407363] Lustre: lustre-MDT0000: new disk, initializing [ 8173.492812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8173.525142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8178.264173] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8189.741995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8189.905966] Lustre: 194373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8189.968454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8189.972664] Lustre: Skipped 1 previous similar message [ 8190.064676] Lustre: lustre-MDT0001: new disk, initializing [ 8190.151755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8190.214110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8190.223552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8195.621931] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8208.248673] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8213.970980] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8221.349897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8221.631643] Lustre: lustre-OST0000: new disk, initializing [ 8221.635165] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8221.640864] Lustre: 196340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8221.753903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8230.483146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8230.495372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8230.647307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8230.820335] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8242.558820] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8248.205958] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8248.486595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8253.326681] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8253.492623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8256.480472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8256.481687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8256.485919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8256.509049] Lustre: Skipped 1 previous similar message [ 8261.644462] Lustre: server umount lustre-OST0000 complete [ 8266.725706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8266.745556] Lustre: Skipped 1 previous similar message [ 8272.397324] Lustre: server umount lustre-MDT0000 complete [ 8276.784556] LustreError: 194364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790904345 with bad export cookie 1288404474745392808 [ 8276.789330] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8276.796793] LustreError: 194364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8277.141703] Lustre: server umount lustre-MDT0001 complete [ 8288.707968] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 21:25:54 (1790904354) [ 8310.791698] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8313.622836] Key type lgssc unregistered [ 8313.890773] LNet: 199292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8313.898445] LNetError: 199292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8314.921516] LNet: Removed LNI 192.168.202.146@tcp [ 8315.847241] Key type .llcrypt unregistered [ 8315.849104] Key type ._llcrypt unregistered [ 8335.584413] Key type ._llcrypt registered [ 8335.587848] Key type .llcrypt registered [ 8335.701629] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8337.119720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8337.397685] alg: No test for adler32 (adler32-zlib) [ 8338.534167] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 8338.808869] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 8340.511383] Key type lgssc registered [ 8341.854906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8350.492488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8350.514377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8352.050951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8356.462978] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8361.446158] LustreError: 200538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8361.473578] LustreError: 200538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8365.095429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8365.457462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8371.178483] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8379.804849] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8386.484708] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8393.841643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8394.064101] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8394.066508] LustreError: 202402:0:(llog.c:1696:llog_backup()) MGC192.168.202.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8394.071201] Lustre: 202402:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8394.140638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8401.932437] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8411.371641] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8415.716763] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8415.723244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8415.737796] Lustre: Skipped 1 previous similar message [ 8415.767852] Lustre: Skipped 2 previous similar messages [ 8420.032427] Lustre: server umount lustre-MDT0000 complete [ 8420.839160] LustreError: 203029:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8420.860318] LustreError: 203029:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8423.346307] LustreError: 200519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790904492 with bad export cookie 3999738012033971369 [ 8423.347534] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8423.363984] LustreError: 200519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8423.767229] Lustre: server umount lustre-MDT0001 complete [ 8438.705739] Lustre: server umount lustre-OST0000 complete [ 8450.659788] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 8464.711813] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8506.775713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8507.006750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8507.033206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8507.119752] Lustre: lustre-MDT0000: new disk, initializing [ 8507.202445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8507.229826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8511.448308] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8522.275585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8522.411616] Lustre: 205928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8522.443166] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8522.445467] Lustre: Skipped 1 previous similar message [ 8522.448568] LustreError: 206710:0:(llog.c:1696:llog_backup()) MGC192.168.202.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8522.453921] LustreError: 206710:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8522.457806] Lustre: 206710:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8522.470095] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8522.473626] Lustre: Skipped 1 previous similar message [ 8522.544838] Lustre: lustre-MDT0001: new disk, initializing [ 8522.664892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8522.701129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8522.708143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8527.200171] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8538.546875] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8544.814185] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8551.623969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8551.865642] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8551.867911] Lustre: Skipped 1 previous similar message [ 8551.870995] LustreError: 207872:0:(llog.c:1696:llog_backup()) MGC192.168.202.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8551.878375] LustreError: 207872:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8551.885734] Lustre: 207872:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8551.892565] Lustre: 207872:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8551.913927] Lustre: lustre-OST0000: new disk, initializing [ 8551.917859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8551.922991] Lustre: 207895:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8552.025442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8557.619458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8557.635095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8557.815286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8558.750530] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8569.598874] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8575.630116] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8575.915112] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8580.871896] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8581.101722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8583.651160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8583.653438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8583.675115] Lustre: Skipped 1 previous similar message [ 8583.678572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8583.716344] Lustre: Skipped 1 previous similar message [ 8588.769736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8588.779095] Lustre: Skipped 1 previous similar message [ 8589.988636] Lustre: server umount lustre-OST0000 complete [ 8599.012698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8599.013895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8599.040655] Lustre: Skipped 2 previous similar messages [ 8599.084922] Lustre: Skipped 1 previous similar message [ 8604.129033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8604.137664] Lustre: Skipped 1 previous similar message [ 8609.251223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8609.469587] Lustre: server umount lustre-MDT0000 complete [ 8613.346802] LustreError: 205939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8613.376022] LustreError: 205939:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8613.797401] LustreError: 205919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790904682 with bad export cookie 3999738012033972090 [ 8613.807275] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8613.814570] LustreError: 205919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8614.137473] Lustre: server umount lustre-MDT0001 complete [ 8623.764487] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 21:31:30 (1790904690) [ 8642.954657] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8653.340808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8653.715516] LustreError: 211025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8653.782639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8657.892160] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8658.916319] LustreError: 211026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8666.828026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8671.714487] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8674.490726] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8680.901750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8686.969712] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8695.633466] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8695.877974] Lustre: lustre-OST0001: new disk, initializing [ 8695.888521] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8695.897515] Lustre: 213334:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8696.001639] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8696.006451] Lustre: Skipped 2 previous similar messages [ 8702.380454] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8706.202880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8706.215405] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8706.392648] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8713.846722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8719.870342] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8721.876274] Lustre: DEBUG MARKER: ADD OST3 [ 8727.285828] loop: module loaded [ 8727.404420] loop0: detected capacity change from 0 to 204800000 [ 8727.600611] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8727.616962] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8727.647865] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8734.030200] loop0: detected capacity change from 0 to 204800000 [ 8735.021913] Lustre: DEBUG MARKER: STOP OST3 [ 8740.887553] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8741.117281] Lustre: lustre-OST0003: new disk, initializing [ 8741.121978] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8741.129373] Lustre: 215334:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8742.377127] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8742.503138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8742.512437] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8742.552355] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8744.416078] LustreError: 215460:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8744.985348] Lustre: Failing over lustre-OST0003 [ 8745.227122] Lustre: server umount lustre-OST0003 complete [ 8747.504273] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8747.533257] LustreError: 212514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8747.549426] LustreError: 212514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8747.647495] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8756.800570] Lustre: DEBUG MARKER: START OST3 again [ 8763.979440] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8764.269512] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8764.279111] Lustre: Skipped 1 previous similar message [ 8764.307380] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8766.057673] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8766.301415] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8766.308439] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8770.618296] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8792.741641] Lustre: Failing over lustre-OST0003 [ 8792.911856] Lustre: server umount lustre-OST0003 complete [ 8795.135566] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8795.166574] LustreError: 217456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8795.176480] Lustre: Skipped 2 previous similar messages [ 8795.214939] LustreError: 217456:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 8815.601417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8815.603182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8815.610303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8815.610312] Lustre: Skipped 2 previous similar messages [ 8815.637393] Lustre: Skipped 3 previous similar messages [ 8815.692411] LustreError: Skipped 1 previous similar message [ 8819.141680] Lustre: server umount lustre-MDT0000 complete [ 8823.195419] LustreError: 211007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790904892 with bad export cookie 3999738012033972965 [ 8823.205076] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8828.835915] LustreError: 213144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8828.868297] LustreError: 213144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 8829.927903] Lustre: server umount lustre-MDT0001 complete [ 8834.811618] Lustre: server umount lustre-OST0000 complete [ 8841.043962] Lustre: server umount lustre-OST0001 complete [ 8849.829400] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 8862.427683] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8905.613634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8905.817791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8905.841105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8905.900810] Lustre: lustre-MDT0000: new disk, initializing [ 8905.956233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8905.969291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8911.545582] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8925.248470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8925.389288] Lustre: 220796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8925.459301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8925.464366] Lustre: Skipped 1 previous similar message [ 8925.545631] Lustre: lustre-MDT0001: new disk, initializing [ 8925.779920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8925.796391] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8930.420288] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8942.127555] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8949.750862] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8956.290251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8956.532597] Lustre: 222768:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8958.104527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8958.133754] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8958.266027] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8963.607177] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8974.323906] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8979.671375] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8979.934206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8985.372550] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8985.639743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8989.167103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8989.169220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8989.181099] LustreError: Skipped 1 previous similar message [ 8989.196687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8989.197951] Lustre: Skipped 3 previous similar messages [ 8989.231986] Lustre: Skipped 8 previous similar messages [ 8993.808746] Lustre: server umount lustre-OST0000 complete [ 9004.644806] Lustre: server umount lustre-MDT0000 complete [ 9009.170076] LustreError: 222767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790905078 with bad export cookie 3999738012033975268 [ 9009.173601] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9009.188129] LustreError: 222767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9009.722943] Lustre: server umount lustre-MDT0001 complete [ 9014.088644] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:38:00 (1790905080) [ 9034.454816] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 9048.969623] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9091.960791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9092.279863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9092.328966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9092.338871] Lustre: Skipped 1 previous similar message [ 9092.443731] Lustre: lustre-MDT0000: new disk, initializing [ 9092.451861] Lustre: Skipped 1 previous similar message [ 9092.569060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9097.458804] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9108.736477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9108.940393] Lustre: 227383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9109.315539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9114.977951] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9127.792604] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9137.530638] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9148.611429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9148.971791] Lustre: 229354:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9150.289724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9150.392891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9155.807507] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9166.096317] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9171.255605] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9171.464423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9176.349751] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9176.530897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9181.153334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9181.162558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9181.171450] LustreError: Skipped 1 previous similar message [ 9181.178131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9181.191082] Lustre: Skipped 3 previous similar messages [ 9181.202475] Lustre: Skipped 5 previous similar messages [ 9184.970827] Lustre: server umount lustre-OST0000 complete [ 9191.394361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9194.987585] Lustre: server umount lustre-MDT0000 complete [ 9196.525936] LustreError: 228258:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9196.551824] LustreError: 228258:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9199.438882] LustreError: 227374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790905268 with bad export cookie 3999738012033976150 [ 9199.446142] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9199.448094] LustreError: 227374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9199.842458] Lustre: server umount lustre-MDT0001 complete [ 9212.607257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9213.469631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9213.473356] Lustre: Skipped 5 previous similar messages [ 9219.471814] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9231.035256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9239.186508] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9249.298788] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9258.097634] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9269.781231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9277.561652] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9287.587126] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9299.425148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9299.429055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9299.432531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9299.438388] Lustre: Skipped 3 previous similar messages [ 9299.444204] Lustre: Skipped 3 previous similar messages [ 9303.918729] Lustre: server umount lustre-OST0000 complete [ 9315.209823] Lustre: server umount lustre-MDT0000 complete [ 9320.031228] LustreError: 231597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790905388 with bad export cookie 3999738012033977032 [ 9320.034599] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9320.049892] LustreError: 231597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9320.627521] Lustre: server umount lustre-MDT0001 complete [ 9428.276928] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 9431.980859] Key type lgssc unregistered [ 9432.538435] LNet: 235240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9432.553472] LNetError: 235240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9432.573567] LNet: Removed LNI 192.168.202.146@tcp [ 9433.824171] Key type .llcrypt unregistered [ 9433.826067] Key type ._llcrypt unregistered [ 9465.656136] Key type ._llcrypt registered [ 9465.660710] Key type .llcrypt registered [ 9465.831874] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 9485.681382] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9487.327880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9487.520430] alg: No test for adler32 (adler32-zlib) [ 9488.664598] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 9488.950810] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 9490.647235] Key type lgssc registered [ 9492.323339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9536.129502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9536.234781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9537.789953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9537.880697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9538.031436] Lustre: lustre-MDT0000: new disk, initializing [ 9538.164979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9538.191713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9542.242844] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9557.609033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9557.737092] Lustre: 238609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9557.784826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9557.787280] Lustre: Skipped 1 previous similar message [ 9557.824945] Lustre: lustre-MDT0001: new disk, initializing [ 9557.907230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9557.937562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9557.955237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9563.188675] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9576.345206] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9583.167414] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9591.288589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9591.662498] Lustre: lustre-OST0000: new disk, initializing [ 9591.667361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9591.681944] Lustre: 240578:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9591.814805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9598.221515] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9600.633830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9600.643613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9600.748466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9609.367462] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9615.667206] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9615.913486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9622.177733] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9622.481220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9626.621421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9626.642946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9626.649544] Lustre: Skipped 1 previous similar message [ 9626.660779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9631.871843] Lustre: server umount lustre-OST0000 complete [ 9636.832057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9636.837302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9636.850040] Lustre: Skipped 1 previous similar message [ 9636.866436] Lustre: Skipped 2 previous similar messages [ 9641.952446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9641.961679] Lustre: Skipped 1 previous similar message [ 9644.325934] Lustre: server umount lustre-MDT0000 complete [ 9647.080706] LustreError: 241921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9647.100672] LustreError: 241921:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9648.658691] LustreError: 238601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790905717 with bad export cookie 3893075837607932170 [ 9648.660935] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9648.670035] LustreError: 238601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9648.952233] Lustre: server umount lustre-MDT0001 complete [ 9675.908799] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9688.283172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9688.968149] LustreError: 243548:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9689.073054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9694.191085] LustreError: 243549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9694.212771] LustreError: 243549:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9695.746594] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9698.272649] LustreError: 243548:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9703.393519] LustreError: 243549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9706.154928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9706.748541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9711.667598] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9715.330416] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9724.305335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9724.812059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9732.247965] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9745.022990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9745.195875] Lustre: lustre-OST0001: new disk, initializing [ 9745.200597] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9745.205468] Lustre: 245856:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9745.269914] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9753.160988] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9754.248659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9754.258917] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9754.395786] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9765.101150] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9773.438563] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9787.818957] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:50:54 (1790905854) [ 9790.435198] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9790.445524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9790.454161] Lustre: Skipped 2 previous similar messages [ 9790.458177] Lustre: Skipped 1 previous similar message [ 9795.555303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9795.558870] Lustre: Skipped 5 previous similar messages [ 9795.918479] Lustre: server umount lustre-MDT0000 complete [ 9799.132777] LustreError: 244283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790905868 with bad export cookie 3893075837607933052 [ 9799.138335] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9799.351827] Lustre: server umount lustre-MDT0001 complete [ 9813.822568] Lustre: server umount lustre-OST0000 complete [ 9816.543308] Lustre: 236824:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790905869/real 1790905869] req@ffff8a06e6f85880 x1877900585454848/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790905885 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9816.572476] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9816.591123] Lustre: Skipped 1 previous similar message [ 9817.523406] Lustre: server umount lustre-OST0001 complete [ 9824.510540] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 9834.791793] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9878.870520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9879.237627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9879.273352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9879.370979] Lustre: lustre-MDT0000: new disk, initializing [ 9879.485464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9879.514550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9885.932804] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9899.404406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9899.502172] Lustre: 249799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9899.538615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9899.541239] Lustre: Skipped 1 previous similar message [ 9899.617555] Lustre: lustre-MDT0001: new disk, initializing [ 9899.729619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9899.754513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9903.829200] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9913.444877] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9919.848480] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9927.061716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9927.314449] Lustre: lustre-OST0000: new disk, initializing [ 9927.318565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9927.323880] Lustre: 251767:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9929.030845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9929.038632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9929.157206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9933.350575] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9943.769915] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9947.990682] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9948.202661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9952.990344] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9953.147045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9959.908658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9959.912177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9959.926477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9959.933090] Lustre: Skipped 1 previous similar message [ 9961.373864] Lustre: server umount lustre-OST0000 complete [ 9979.359152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9979.371031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9979.382494] Lustre: Skipped 8 previous similar messages [ 9979.581725] Lustre: server umount lustre-MDT0000 complete [ 9983.137934] LustreError: 249791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790906052 with bad export cookie 3893075837607935187 [ 9983.157219] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9983.166433] LustreError: 249791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9983.450506] Lustre: server umount lustre-MDT0001 complete [ 9991.876085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9992.630845] LustreError: 253990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9992.773768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9992.792065] Lustre: Skipped 2 previous similar messages [ 9997.431613] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10006.264606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10012.756489] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10022.445135] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10028.769366] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10036.252422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10044.358610] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10052.930504] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10200.548161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10200.549202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10200.550301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10200.566870] Lustre: Skipped 3 previous similar messages [10200.583347] LustreError: Skipped 1 previous similar message [10204.563493] Lustre: server umount lustre-OST0000 complete [10214.610358] Lustre: server umount lustre-MDT0000 complete [10215.914663] LustreError: 255173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10215.934145] LustreError: 255173:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10217.869321] LustreError: 253972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790906286 with bad export cookie 3893075837607936069 [10217.876189] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10217.877100] LustreError: 253972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10218.112596] Lustre: server umount lustre-MDT0001 complete [10323.160825] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [10325.440625] Key type lgssc unregistered [10325.694222] LNet: 257601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10325.712946] LNetError: 257601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10325.739984] LNet: Removed LNI 192.168.202.146@tcp [10326.640514] Key type .llcrypt unregistered [10326.642307] Key type ._llcrypt unregistered [10352.550303] Key type ._llcrypt registered [10352.552685] Key type .llcrypt registered [10352.662459] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [10368.355383] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [10369.096463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10369.187045] alg: No test for adler32 (adler32-zlib) [10370.749250] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [10371.090606] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [10372.807210] Key type lgssc registered [10373.711875] Lustre: Echo OBD driver; http://www.lustre.org/ [10420.035245] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [10430.984934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10431.047573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10432.409295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10432.429565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10432.514677] Lustre: lustre-MDT0000: new disk, initializing [10432.589344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10432.605424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10437.896399] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10450.694456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10450.926293] Lustre: 261697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10451.052448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10451.074508] Lustre: Skipped 1 previous similar message [10451.284180] Lustre: lustre-MDT0001: new disk, initializing [10451.506175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10451.579296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10451.586768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10456.858232] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10461.791782] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10468.353799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10468.872629] Lustre: lustre-OST0000: new disk, initializing [10468.882183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10468.890206] Lustre: 263331:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10469.053838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10475.108032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10475.124702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10475.342401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10477.514979] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10491.409660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10491.581945] Lustre: lustre-OST0001: new disk, initializing [10491.585825] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10491.591442] Lustre: 264202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10491.693236] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10497.641064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10497.649816] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10497.776146] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10499.045912] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10511.864474] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10520.698181] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10538.483786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10538.487081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10538.506572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10538.988692] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10539.000382] Lustre: Skipped 2 previous similar messages [10539.010750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10539.018184] Lustre: Skipped 2 previous similar messages [10540.569058] Lustre: server umount lustre-MDT0000 complete [10544.749846] LustreError: 261689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790906613 with bad export cookie 9590785762983085094 [10544.761184] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10544.764554] LustreError: 261689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10545.206491] Lustre: server umount lustre-MDT0001 complete [10559.974293] Lustre: server umount lustre-OST0000 complete [10574.760751] Lustre: server umount lustre-OST0001 complete [10589.736558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10599.335904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10607.669929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10618.630894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10639.449519] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [10650.628684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10650.642685] Lustre: lustre-MDT0000: reset Object Index mappings [10650.930302] LustreError: 269214:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10650.942151] LustreError: 269214:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10650.996834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10654.991085] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10656.225325] LustreError: 269215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10660.322379] LustreError: 269214:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10663.953091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10663.983497] Lustre: lustre-MDT0001: reset Object Index mappings [10664.514336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10669.946351] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10673.225701] Lustre: 270356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10679.864143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10686.047621] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10689.456866] LustreError: 271237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10694.629648] LustreError: 270709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10695.679856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10695.832923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10696.218759] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10696.221333] Lustre: Skipped 1 previous similar message [10701.321146] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10702.336937] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10709.368782] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10713.085731] Lustre: 272227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10720.740349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10720.758200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10720.780745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10725.448686] Lustre: server umount lustre-MDT0000 complete [10726.900485] LustreError: 271517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10726.928767] LustreError: 271517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10729.626901] LustreError: 269196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790906798 with bad export cookie 9590785762983091877 [10729.632728] LustreError: 269196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10729.640692] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10730.014556] Lustre: server umount lustre-MDT0001 complete [10744.815179] Lustre: server umount lustre-OST0000 complete [10747.692331] Lustre: 259184:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790906800/real 1790906800] req@ffff8a06f5034000 x1877901510315136/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790906816 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10747.733792] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10747.749242] Lustre: Skipped 3 previous similar messages [10749.209764] Lustre: server umount lustre-OST0001 complete [10774.398825] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 22:07:20 (1790906840) [10793.782969] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [10806.208755] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [10840.374424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10840.566143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10840.603149] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10840.690656] Lustre: lustre-MDT0000: new disk, initializing [10840.764419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10840.780356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10845.257043] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10856.214623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10856.317551] Lustre: 276481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10856.363375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10856.366508] Lustre: Skipped 1 previous similar message [10856.419182] Lustre: lustre-MDT0001: new disk, initializing [10856.548936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10856.564143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10860.817664] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10869.803342] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10876.589739] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10885.620640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10886.171204] Lustre: lustre-OST0000: new disk, initializing [10886.189697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10886.198955] Lustre: 278451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10887.794345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10887.808840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10887.992628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10895.096350] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10905.314876] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10909.894600] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10910.081383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10914.322968] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10914.528128] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10918.881915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.899212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10918.901855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10918.912588] Lustre: Skipped 5 previous similar messages [10922.455273] Lustre: server umount lustre-OST0000 complete [10928.098804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10928.110226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10928.129568] Lustre: Skipped 1 previous similar message [10928.143805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10932.117139] Lustre: server umount lustre-MDT0000 complete [10934.240731] LustreError: 276494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10934.255252] LustreError: 276494:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10935.719796] LustreError: 276474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790907004 with bad export cookie 9590785762983093977 [10935.728961] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10935.730879] LustreError: 276474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10936.170809] Lustre: server umount lustre-MDT0001 complete [10953.689789] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [10964.641372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10965.241128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10965.249141] Lustre: Skipped 2 previous similar messages [10970.011958] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10970.599150] LustreError: 281386:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10970.625097] LustreError: 281386:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10978.796281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10983.573764] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10986.858515] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10996.029638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11003.692027] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11013.190852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11013.426568] Lustre: lustre-OST0001: new disk, initializing [11013.432059] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11013.438848] Lustre: 283697:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11015.527338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11015.536781] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11015.686923] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11020.846873] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11031.088119] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11035.182209] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11040.653788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11040.703720] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11050.976832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11050.983117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11050.993138] Lustre: Skipped 1 previous similar message [11050.997766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11051.001449] Lustre: Skipped 2 previous similar messages [11052.509639] Lustre: server umount lustre-MDT0000 complete [11056.048170] LustreError: 281368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790907124 with bad export cookie 9590785762983094852 [11056.065033] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11056.401211] Lustre: server umount lustre-MDT0001 complete [11071.126148] Lustre: server umount lustre-OST0000 complete [11072.994322] Lustre: 259184:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790907125/real 1790907125] req@ffff8a05c1e3e300 x1877901510421248/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790907141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11075.345649] Lustre: server umount lustre-OST0001 complete [11093.496388] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [11103.929975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11104.356191] LustreError: 287037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11104.384905] LustreError: 287037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11104.448872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11104.451886] Lustre: Skipped 3 previous similar messages [11109.263084] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11118.498204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11124.362297] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11127.938441] Lustre: 288179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11135.423937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11141.312366] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11149.212522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11150.897650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11156.502771] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11158.002348] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11165.674321] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11169.471861] Lustre: 290054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11174.775894] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11174.844929] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11189.166883] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 22:14:15 (1790907255) [11206.112172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11206.126591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11206.148547] Lustre: Skipped 4 previous similar messages [11206.163709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11206.166690] Lustre: Skipped 4 previous similar messages [11211.114534] Lustre: server umount lustre-MDT0000 complete [11214.712106] LustreError: 288556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790907283 with bad export cookie 9590785762983101292 [11214.719825] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11214.730954] LustreError: 288556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11229.151259] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11229.454385] Lustre: server umount lustre-MDT0001 complete [11239.806301] Lustre: server umount lustre-OST0000 complete [11244.358498] Lustre: server umount lustre-OST0001 complete [11251.720337] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [11262.533461] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [11313.962495] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [11325.825837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11326.156497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11326.196504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11326.294350] Lustre: lustre-MDT0000: new disk, initializing [11326.450941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11331.432940] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11343.498451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11343.634612] Lustre: 294603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11343.868285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11348.487959] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11362.125073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11362.515867] Lustre: 296236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11362.659656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11362.667100] Lustre: Skipped 5 previous similar messages [11364.075711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11369.609501] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11381.158675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11381.360972] Lustre: 297108:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11382.562290] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11382.576156] Lustre: Skipped 1 previous similar message [11382.733068] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11388.711880] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11399.455521] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11403.553306] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11403.562643] Lustre: Skipped 1 previous similar message [11421.469981] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 22:18:08 (1790907488) [11425.762463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11425.773365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11425.800938] Lustre: Skipped 5 previous similar messages [11425.808894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11425.811532] Lustre: Skipped 10 previous similar messages [11426.294196] Lustre: server umount lustre-MDT0000 complete [11428.853948] LustreError: 294609:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11428.890535] LustreError: 294609:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [11431.345335] LustreError: 297107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790907500 with bad export cookie 9590785762983109202 [11431.356034] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11431.370066] LustreError: 297107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11431.914732] Lustre: server umount lustre-MDT0001 complete [11447.852073] Lustre: server umount lustre-OST0000 complete [11449.128065] Lustre: 259184:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790907502/real 1790907502] req@ffff8a06ee5b0000 x1877901510579456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790907518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11451.740778] Lustre: server umount lustre-OST0001 complete [11462.356627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11468.899331] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11479.706868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11485.651862] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11494.309505] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11502.956766] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11510.048143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11517.138304] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11526.218696] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11559.729654] loop1: detected capacity change from 0 to 204800000 [11560.105597] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11560.130035] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11560.158168] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11566.183754] loop1: detected capacity change from 0 to 204800000 [11573.355664] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11573.437713] LustreError: 299342:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs13241: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11573.458333] LustreError: 299342:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs13241 [11573.465965] LustreError: 299342:0:(mgs_handler.c:517:mgs_target_reg()) fs13241-MDT0001: Error checking target index: rc = -13 [11573.474230] LustreError: fs13241-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11573.487700] LustreError: 303100:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11573.494363] LustreError: 303100:0:(tgt_mount.c:2128:server_put_super()) no obd fs13241-MDT0001 [11573.500959] LustreError: 303100:0:(tgt_mount.c:116:server_deregister_mount()) fs13241-MDT0001 not registered [11573.542117] Lustre: server umount fs13241-MDT0001 complete [11573.545051] LustreError: 303100:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11583.041075] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11583.127187] Lustre: Setting parameter fs13241-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs13241-MDT0001 [11583.178268] Lustre: srv-fs13241-MDT0001: No data found on store. Initialize space. [11583.182252] Lustre: Skipped 4 previous similar messages [11583.265703] Lustre: fs13241-MDT0001: new disk, initializing [11583.269492] Lustre: Skipped 3 previous similar messages [11583.393410] Lustre: srv-fs13241-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11583.398901] Lustre: 303566:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13241-MDT0001-mdtlov: can't create new llog: rc = -115 [11583.406080] LustreError: 303566:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13241-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11588.141101] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11595.941336] Lustre: server umount fs13241-MDT0001 complete [11604.969397] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11611.608850] loop1: detected capacity change from 0 to 204800000 [11617.738404] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11618.191244] Lustre: 305135:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13241-MDT0001-mdtlov: can't create new llog: rc = -115 [11618.197432] LustreError: 305135:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13241-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11623.264692] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11628.303566] Lustre: server umount fs13241-MDT0001 complete [11639.265180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11639.279422] Lustre: Skipped 4 previous similar messages [11639.292064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11644.827721] Lustre: server umount lustre-MDT0000 complete [11655.106029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11655.334283] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11660.823402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11661.113353] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11670.823501] loop1: detected capacity change from 0 to 204800000 [11676.954784] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11677.501915] Lustre: 307580:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13241-MDT0001-mdtlov: can't create new llog: rc = -115 [11677.508520] LustreError: 307580:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13241-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11682.045857] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11685.855514] LustreError: 301186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11685.886185] LustreError: 301186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 101 previous similar messages [11687.797314] Lustre: server umount fs13241-MDT0001 complete [11711.977606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11711.985106] LustreError: Skipped 2 previous similar messages [11725.279690] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11725.554876] Lustre: server umount lustre-OST0000 complete [11735.956432] Lustre: server umount lustre-MDT0000 complete [11739.918210] LustreError: 299336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790907808 with bad export cookie 9590785762983111883 [11739.926772] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11739.937057] LustreError: 299336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11740.266849] Lustre: server umount lustre-MDT0001 complete [11751.456381] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [11754.264450] Key type lgssc unregistered [11754.556663] LNet: 309926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11754.559158] LNetError: 309926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11754.569721] LNet: Removed LNI 192.168.202.146@tcp [11755.327167] Key type .llcrypt unregistered [11755.328843] Key type ._llcrypt unregistered [11777.305907] Key type ._llcrypt registered [11777.308870] Key type .llcrypt registered [11777.445314] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [11778.737564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11778.912518] alg: No test for adler32 (adler32-zlib) [11780.235787] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [11780.586299] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [11782.271333] Key type lgssc registered [11783.224695] Lustre: Echo OBD driver; http://www.lustre.org/ [11793.157709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11793.173210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11794.542586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11799.465699] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11804.148670] LustreError: 311214:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11804.189233] LustreError: 311214:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11809.257426] LustreError: 311213:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11810.349553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11811.059693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11816.698947] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11826.561593] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11833.800571] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11841.275284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11841.756131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11849.494302] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11856.118326] LustreError: 313046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11858.998533] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11861.528432] LustreError: 313046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11866.132890] LustreError: 313047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11866.160515] LustreError: 313047:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11874.811643] LustreError: 313667:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11874.834141] LustreError: 313667:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11884.864205] loop1: detected capacity change from 0 to 204800000 [11885.206782] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11885.221400] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11885.238377] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11890.595334] loop1: detected capacity change from 0 to 204800000 [11891.699862] LustreError: 313047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11891.736082] LustreError: 313047:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [11897.782591] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11897.951439] Lustre: fs31146-OST0000: new disk, initializing [11897.954821] Lustre: srv-fs31146-OST0000: No data found on store. Initialize space. [11897.962248] Lustre: 314950:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11898.086447] Lustre: fs31146-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11905.515403] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11916.926435] Lustre: server umount fs31146-OST0000 complete [11926.000816] LustreError: 313044:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11926.030246] LustreError: 313044:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [11932.298699] loop1: detected capacity change from 0 to 204800000 [11934.195685] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11937.704332] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11937.966118] Lustre: fs31146-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11939.326865] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11944.985883] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11956.461728] loop2: detected capacity change from 0 to 204800000 [11956.840573] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11956.867576] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11956.918970] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11963.502123] loop2: detected capacity change from 0 to 204800000 [11971.277392] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11971.411520] LustreError: fs31146-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11971.429977] LustreError: fs31146-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11971.440830] LustreError: 317995:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11971.449795] LustreError: 317995:0:(tgt_mount.c:2128:server_put_super()) no obd fs31146-OST0001 [11971.457537] LustreError: 317995:0:(tgt_mount.c:116:server_deregister_mount()) fs31146-OST0001 not registered [11971.490524] Lustre: server umount fs31146-OST0001 complete [11971.493347] LustreError: 317995:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11981.127389] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11981.317303] Lustre: fs31146-OST0001: new disk, initializing [11981.323492] Lustre: srv-fs31146-OST0001: No data found on store. Initialize space. [11981.333317] Lustre: 318547:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11981.411530] Lustre: fs31146-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11988.314017] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11990.506570] LustreError: 313667:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11990.535539] LustreError: 313667:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [11997.094860] Lustre: server umount fs31146-OST0000 complete [12008.278185] Lustre: server umount fs31146-OST0001 complete [12025.518937] loop1: detected capacity change from 0 to 204800000 [12032.621518] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12032.821897] Lustre: fs31146-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12040.967367] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12049.531515] loop2: detected capacity change from 0 to 204800000 [12055.899777] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12064.885757] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12075.835787] Lustre: server umount fs31146-OST0000 complete [12085.324834] Lustre: server umount fs31146-OST0001 complete [12087.785849] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12092.907314] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12100.071573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12100.080071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12100.105425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12102.114824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12102.137117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12103.138131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12103.163483] Lustre: Skipped 1 previous similar message [12105.849922] Lustre: server umount lustre-OST0000 complete [12112.865320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12112.873607] LustreError: Skipped 1 previous similar message [12112.878215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12112.893469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12116.882724] Lustre: server umount lustre-MDT0000 complete [12120.610199] LustreError: 311192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790908189 with bad export cookie 8223957009246891161 [12120.613945] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12120.629731] LustreError: 311192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12121.004286] Lustre: server umount lustre-MDT0001 complete [12132.469602] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [12135.186968] Key type lgssc unregistered [12135.462539] LNet: 324470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12135.473986] LNetError: 324470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12135.501532] LNet: Removed LNI 192.168.202.146@tcp [12136.389246] Key type .llcrypt unregistered [12136.391469] Key type ._llcrypt unregistered [12172.266671] Key type ._llcrypt registered [12172.270335] Key type .llcrypt registered [12172.442100] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [12192.197991] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [12194.067836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12194.404310] alg: No test for adler32 (adler32-zlib) [12195.691806] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [12196.100124] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [12197.887603] Key type lgssc registered [12199.692609] Lustre: Echo OBD driver; http://www.lustre.org/ [12241.893759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12241.934864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12243.259562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12243.326878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12243.445192] Lustre: lustre-MDT0000: new disk, initializing [12243.550042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12243.566987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12248.604366] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12259.485647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12259.582599] Lustre: 328351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12259.633349] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12259.636769] Lustre: Skipped 1 previous similar message [12259.731358] Lustre: lustre-MDT0001: new disk, initializing [12259.840955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12259.886637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12259.905230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12265.777301] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12277.273201] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12284.124863] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12293.093125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12293.541749] Lustre: lustre-OST0000: new disk, initializing [12293.545220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12293.551074] Lustre: 330320:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12293.666946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12299.321557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12299.335751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12299.472911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12301.004293] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12313.295824] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12316.260992] LustreError: 326512:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [12316.330355] Lustre: Mounted lustre-client - version 2.17.58_63_g3705aed [12317.826880] Lustre: Unmounted lustre-client [12330.465088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12330.476275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12330.499860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12330.506930] Lustre: Skipped 1 previous similar message [12332.019752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12332.033517] Lustre: Skipped 1 previous similar message [12335.651239] Lustre: server umount lustre-OST0000 complete [12341.731916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12341.742429] LustreError: Skipped 1 previous similar message [12341.746602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12341.764775] Lustre: Skipped 1 previous similar message [12341.770561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12347.362197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12347.374729] Lustre: Skipped 3 previous similar messages [12352.489833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12352.500043] Lustre: Skipped 1 previous similar message [12354.015210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12354.226786] Lustre: server umount lustre-MDT0000 complete [12358.436720] LustreError: 329145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790908427 with bad export cookie 10357381859492185663 [12358.449379] LustreError: 329145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12358.453720] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12358.957417] Lustre: server umount lustre-MDT0001 complete [12373.539771] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [12377.669053] Key type lgssc unregistered [12378.976625] LNet: 332482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12378.993115] LNetError: 332482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12445.608778] LNet: Removed LNI 192.168.202.146@tcp [12446.713286] Key type .llcrypt unregistered [12446.715123] Key type ._llcrypt unregistered [12466.361437] Key type ._llcrypt registered [12466.363020] Key type .llcrypt registered [12466.491074] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [12485.469375] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [12486.806054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12486.863323] alg: No test for adler32 (adler32-zlib) [12487.867700] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [12488.168488] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [12489.863278] Key type lgssc registered [12490.816941] Lustre: Echo OBD driver; http://www.lustre.org/ [12527.913864] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 22:36:33 (1790908593) [12537.089328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12537.152812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12538.402701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12538.454689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12538.634512] Lustre: lustre-MDT0000: new disk, initializing [12538.746733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12538.774363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12543.795993] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12554.523301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12554.657655] Lustre: 335717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12554.703668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12554.707359] Lustre: Skipped 1 previous similar message [12554.769124] Lustre: lustre-MDT0001: new disk, initializing [12554.866090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12554.920647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12554.929325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12558.585452] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12567.670791] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12573.102852] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12582.099393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12582.373387] Lustre: lustre-OST0000: new disk, initializing [12582.379332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12582.385846] Lustre: 337688:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12582.528790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12588.647301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12588.663416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12588.896458] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12591.231488] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12601.621504] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12619.751344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12619.752991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12619.754817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12619.768183] Lustre: Skipped 1 previous similar message [12624.690635] Lustre: server umount lustre-OST0000 complete [12629.985551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12629.988243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12630.007427] Lustre: Skipped 1 previous similar message [12630.041791] Lustre: Skipped 2 previous similar messages [12635.108295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12635.130955] Lustre: Skipped 1 previous similar message [12639.228600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12643.295143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12643.843428] Lustre: server umount lustre-MDT0000 complete [12648.181245] LustreError: 335709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790908717 with bad export cookie 1106677535757551460 [12648.194457] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12648.195478] LustreError: 335709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12648.698931] Lustre: server umount lustre-MDT0001 complete [12663.961869] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [12667.940960] Key type lgssc unregistered [12668.449655] LNet: 339598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12668.458605] LNetError: 339598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12668.478722] LNet: Removed LNI 192.168.202.146@tcp [12669.938815] Key type .llcrypt unregistered [12669.940355] Key type ._llcrypt unregistered [12695.421655] Key type ._llcrypt registered [12695.424324] Key type .llcrypt registered [12695.545839] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [12696.763453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12696.946476] alg: No test for adler32 (adler32-zlib) [12698.400887] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [12698.811728] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [12700.567255] Key type lgssc registered [12702.154938] Lustre: Echo OBD driver; http://www.lustre.org/ [12711.326798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12711.356717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12712.830370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12718.204187] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12722.151840] LustreError: 340888:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12722.194378] LustreError: 340888:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12727.265984] LustreError: 340887:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12729.294687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12729.813330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12735.095799] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12744.882396] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12752.632055] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12760.731761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12761.020405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12769.438466] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12779.063742] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12797.548605] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12799.774638] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12806.640980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12806.652292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12806.684041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12806.691657] Lustre: Skipped 1 previous similar message [12808.161709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12808.165337] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12808.191674] Lustre: Skipped 1 previous similar message [12810.935939] Lustre: server umount lustre-MDT0000 complete [12811.753532] LustreError: 343144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12816.766833] LustreError: 340869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790908885 with bad export cookie 17632350713252090414 [12816.795385] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12816.873300] LustreError: 340884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12816.896352] LustreError: 340884:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12816.897800] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.46@tcp (stopping) [12816.915710] Lustre: Skipped 2 previous similar messages [12818.401569] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12821.408714] LustreError: 340882:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12821.409237] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12821.440244] LustreError: 340882:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12821.461281] Lustre: Skipped 1 previous similar message [12826.529462] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12826.547736] Lustre: Skipped 1 previous similar message [12831.199186] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12831.690415] Lustre: server umount lustre-MDT0001 complete [12836.784514] Lustre: server umount lustre-OST0000 complete [12847.077231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12852.922546] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12857.137592] Lustre: 345089:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.146@tcp [12859.809346] Lustre: server umount MGS complete [12870.377757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12871.215654] LustreError: 345662:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12871.235475] LustreError: 345662:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12871.400226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12876.783915] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12885.807350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12886.122782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12890.570420] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12901.283107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12901.921816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12908.753846] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12917.475919] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12919.473603] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12926.912552] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12929.092472] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12936.683050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12936.697151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12936.711166] LustreError: Skipped 1 previous similar message [12936.713479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12936.725584] Lustre: Skipped 1 previous similar message [12936.748608] Lustre: Skipped 2 previous similar messages [12941.830718] Lustre: server umount lustre-OST0000 complete [12946.918115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12946.942153] Lustre: Skipped 1 previous similar message [12952.843055] Lustre: server umount lustre-MDT0000 complete [12956.139063] LustreError: 345663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12956.159433] LustreError: 345663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12957.827169] LustreError: 345642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790909026 with bad export cookie 17632350713252091261 [12957.829422] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12957.836414] LustreError: 345642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12958.429523] Lustre: server umount lustre-MDT0001 complete [12973.967815] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [12977.460512] Key type lgssc unregistered [12977.762452] LNet: 349100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12977.768057] LNetError: 349100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12977.786083] LNet: Removed LNI 192.168.202.146@tcp [12978.837158] Key type .llcrypt unregistered [12978.839656] Key type ._llcrypt unregistered [13001.342865] Key type ._llcrypt registered [13001.344659] Key type .llcrypt registered [13001.499051] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [13018.352616] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [13019.612254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13019.780887] alg: No test for adler32 (adler32-zlib) [13021.058264] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [13021.359147] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [13023.015237] Key type lgssc registered [13024.265825] Lustre: Echo OBD driver; http://www.lustre.org/ [13073.703807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13073.745796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13075.010755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13075.054873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13075.204349] Lustre: lustre-MDT0000: new disk, initializing [13075.292411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13075.314490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13080.483215] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13092.893635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13093.095090] Lustre: 352279:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13093.174431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13093.180322] Lustre: Skipped 1 previous similar message [13093.304739] Lustre: lustre-MDT0001: new disk, initializing [13093.462059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13093.517755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13093.525319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13100.963283] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13112.270468] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13120.188340] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13128.395511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13128.851520] Lustre: lustre-OST0000: new disk, initializing [13128.866592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13128.876697] Lustre: 354253:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13129.003825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13137.214896] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13137.516933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13137.543241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13137.777232] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13148.216371] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13154.401333] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13154.803798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13159.874124] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13160.113487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13163.502030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13163.515033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13163.539147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13163.549321] LustreError: Skipped 1 previous similar message [13168.612251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13168.626580] Lustre: Skipped 2 previous similar messages [13173.746613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13176.287235] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13176.577815] Lustre: server umount lustre-OST0000 complete [13183.972041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13183.981036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13183.987916] Lustre: Skipped 2 previous similar messages [13184.000333] Lustre: Skipped 2 previous similar messages [13187.335055] Lustre: server umount lustre-MDT0000 complete [13191.108477] LustreError: 354252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790909259 with bad export cookie 6936605490251687289 [13191.110974] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13191.120870] LustreError: 354252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13191.481308] Lustre: server umount lustre-MDT0001 complete [13201.666738] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 22:47:47 (1790909267) [13220.183703] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [13233.768831] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [13277.736870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13278.022048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13278.107194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13278.243385] Lustre: lustre-MDT0000: new disk, initializing [13278.338226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13278.372339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13283.943502] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13295.963609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13296.185631] Lustre: 359015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13296.268354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13296.272530] Lustre: Skipped 1 previous similar message [13296.381733] Lustre: lustre-MDT0001: new disk, initializing [13296.498906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13296.571171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13296.588168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13302.008449] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13312.879676] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13319.550786] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13328.076749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13328.516159] Lustre: lustre-OST0000: new disk, initializing [13328.519958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13328.525665] Lustre: 360985:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13328.617925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13334.601618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13334.605737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13334.713703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13335.744613] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13345.295273] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13386.213134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13386.218957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13386.234815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13386.241545] Lustre: Skipped 1 previous similar message [13386.249187] Lustre: Skipped 1 previous similar message [13389.778637] Lustre: server umount lustre-OST0000 complete [13396.451645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13396.492956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13396.507514] Lustre: Skipped 1 previous similar message [13400.775500] Lustre: server umount lustre-MDT0000 complete [13401.569082] LustreError: 359028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13401.605023] LustreError: 359028:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13404.840121] LustreError: 359008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790909473 with bad export cookie 6936605490251688171 [13404.850766] LustreError: 359008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13404.852726] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13405.176754] Lustre: server umount lustre-MDT0001 complete [13417.245193] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13420.255202] Key type lgssc unregistered [13420.537793] LNet: 363158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13420.544271] LNetError: 363158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13421.614266] LNet: Removed LNI 192.168.202.146@tcp [13422.574825] Key type .llcrypt unregistered [13422.584136] Key type ._llcrypt unregistered [13535.745181] Key type ._llcrypt registered [13535.747634] Key type .llcrypt registered [13535.850240] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13536.143416] Key type .llcrypt unregistered [13536.146484] Key type ._llcrypt unregistered [13628.004963] Key type ._llcrypt registered [13628.008471] Key type .llcrypt registered [13628.146976] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13628.624297] Key type .llcrypt unregistered [13628.629251] Key type ._llcrypt unregistered [13669.187512] Key type ._llcrypt registered [13669.195712] Key type .llcrypt registered [13669.421744] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13669.903780] Key type .llcrypt unregistered [13669.907252] Key type ._llcrypt unregistered [13710.179459] Key type ._llcrypt registered [13710.181467] Key type .llcrypt registered [13710.275442] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13710.520337] Key type .llcrypt unregistered [13710.521910] Key type ._llcrypt unregistered [13807.132792] Key type ._llcrypt registered [13807.137541] Key type .llcrypt registered [13807.252536] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13807.580240] Key type .llcrypt unregistered [13807.582552] Key type ._llcrypt unregistered [13843.352892] Key type ._llcrypt registered [13843.354379] Key type .llcrypt registered [13843.444556] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13843.709457] Key type .llcrypt unregistered [13843.714874] Key type ._llcrypt unregistered [13877.709985] Key type ._llcrypt registered [13877.712422] Key type .llcrypt registered [13877.803866] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13878.014856] Key type .llcrypt unregistered [13878.018522] Key type ._llcrypt unregistered [13912.703422] Key type ._llcrypt registered [13912.709412] Key type .llcrypt registered [13912.843537] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13913.244320] Key type .llcrypt unregistered [13913.246091] Key type ._llcrypt unregistered [13954.390311] Key type ._llcrypt registered [13954.392265] Key type .llcrypt registered [13954.538182] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13954.873597] Key type .llcrypt unregistered [13954.877758] Key type ._llcrypt unregistered [13995.639833] Key type ._llcrypt registered [13995.643562] Key type .llcrypt registered [13995.778473] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13996.218592] Key type .llcrypt unregistered [13996.220355] Key type ._llcrypt unregistered [14072.744626] Key type ._llcrypt registered [14072.747992] Key type .llcrypt registered [14072.961980] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14073.349321] Key type .llcrypt unregistered [14073.351820] Key type ._llcrypt unregistered [14114.526633] Key type ._llcrypt registered [14114.529427] Key type .llcrypt registered [14114.682960] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14115.031823] Key type .llcrypt unregistered [14115.033608] Key type ._llcrypt unregistered [14153.195942] Key type ._llcrypt registered [14153.199274] Key type .llcrypt registered [14153.328788] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14153.714566] Key type .llcrypt unregistered [14153.720147] Key type ._llcrypt unregistered [14217.427334] Key type ._llcrypt registered [14217.433280] Key type .llcrypt registered [14217.532381] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [14237.934730] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [14239.462529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14239.637423] alg: No test for adler32 (adler32-zlib) [14240.869791] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [14241.313966] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [14243.111650] Key type lgssc registered [14244.540808] Lustre: Echo OBD driver; http://www.lustre.org/ [14288.893526] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14288.962159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14290.373989] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14290.394510] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14290.457775] Lustre: tst250fs-MDT0000: new disk, initializing [14290.513109] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14290.529659] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14294.690984] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [14305.908117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14306.034412] Lustre: 383536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14306.087140] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14306.089799] Lustre: Skipped 1 previous similar message [14306.130422] Lustre: tst250fs-MDT0001: new disk, initializing [14306.210557] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14306.242941] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14306.257529] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14310.416434] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [14315.778378] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14323.681953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14323.922550] Lustre: tst250fs-OST0000: new disk, initializing [14323.926192] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14323.928924] Lustre: 385168:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14324.031118] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14331.499954] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14331.517830] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14331.636581] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [14331.688515] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14342.821194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14342.984593] Lustre: tst250fs-OST0001: new disk, initializing [14342.990594] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14342.998235] Lustre: 386037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14343.103171] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14349.503791] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14349.512441] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14349.586775] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14351.282763] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [14357.205971] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14360.183246] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14360.189675] Lustre: Skipped 2 previous similar messages [14383.074915] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14383.085275] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14383.097722] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14385.656470] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14385.662496] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14389.696736] Lustre: server umount tst250fs-MDT0000 complete [14394.448819] LustreError: 383528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790910463 with bad export cookie 17679541814157658358 [14394.455271] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14394.459835] LustreError: 383528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14394.789668] Lustre: server umount tst250fs-MDT0001 complete [14410.071502] Lustre: server umount tst250fs-OST0000 complete [14411.240655] Lustre: 381751:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790910464/real 1790910464] req@ffff8a06f4f22a00 x1877905569294336/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790910480 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14411.282371] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14411.297739] Lustre: Skipped 2 previous similar messages [14414.783602] Lustre: server umount tst250fs-OST0001 complete [14435.239380] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [14444.754128] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [14481.497768] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 23:09:07 (1790910547) [14484.035294] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14506.970841] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [14520.651411] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [14559.842520] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14222 sec ========= 23:10:26 (1790910626) [14562.405626] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:10:28 (1790910628) === [14566.708526] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:10:33 (1790910633) === [14615.334560] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14619.145462] Key type lgssc unregistered [14619.580547] LNet: 395189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14619.586736] LNetError: 395189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14620.644943] LNet: Removed LNI 192.168.202.146@tcp [14621.718862] Key type .llcrypt unregistered [14621.728202] Key type ._llcrypt unregistered