[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 652104836 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002425] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004030] kvm-guest: setup PV IPIs [ 0.009545] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011012] pid_max: default: 32768 minimum: 301 [ 0.012128] LSM: Security Framework initializing [ 0.013058] Yama: becoming mindful. [ 0.014032] SELinux: Initializing. [ 0.015057] *** VALIDATE selinux *** [ 0.026274] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.033395] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.035000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036107] *** VALIDATE tmpfs *** [ 0.038065] *** VALIDATE proc *** [ 0.039000] *** VALIDATE cgroup *** [ 0.039011] *** VALIDATE cgroup2 *** [ 0.040257] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041173] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043030] Spectre V2 : User space: Vulnerable [ 0.044031] Speculative Store Bypass: Vulnerable [ 0.047341] debug: unmapping init [mem 0xffffffffa2259000-0xffffffffa2260fff] [ 0.050272] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051733] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052028] ... version: 2 [ 0.053010] ... bit width: 48 [ 0.054014] ... generic registers: 4 [ 0.055011] ... value mask: 0000ffffffffffff [ 0.056018] ... max period: 00007fffffffffff [ 0.057010] ... fixed-purpose events: 3 [ 0.058012] ... event mask: 000000070000000f [ 0.059267] rcu: Hierarchical SRCU implementation. [ 0.061552] smp: Bringing up secondary CPUs ... [ 0.063393] x86: Booting SMP configuration: [ 0.064024] .... node #0, CPUs: #1 #2 #3 [ 0.072183] smp: Brought up 1 node, 4 CPUs [ 0.074014] smpboot: Max logical packages: 1 [ 0.075013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.120214] node 0 deferred pages initialised in 43ms [ 0.126094] devtmpfs: initialized [ 0.127317] x86/mm: Memory block size: 128MB [ 0.130807] gcov: version magic: 0x41383552 [ 0.135291] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137363] pinctrl core: initialized pinctrl subsystem [ 0.139200] [ 0.139721] ************************************************************* [ 0.141009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143008] ** ** [ 0.145010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.146007] ** ** [ 0.148008] ** This means that this kernel is built to expose internal ** [ 0.150008] ** IOMMU data structures, which may compromise security on ** [ 0.152008] ** your system. ** [ 0.154008] ** ** [ 0.155007] ** If you see this message and you are not debugging the ** [ 0.157009] ** kernel, report this immediately to your vendor! ** [ 0.159009] ** ** [ 0.161010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163009] ************************************************************* [ 0.165631] NET: Registered protocol family 16 [ 0.167373] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174044] cpuidle: using governor menu [ 0.175564] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177455] PCI: Using configuration type 1 for base access [ 0.179137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.187063] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.189110] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192247] cryptd: max_cpu_qlen set to 1000 [ 0.193164] ACPI: Added _OSI(Module Device) [ 0.194000] ACPI: Added _OSI(Processor Device) [ 0.194007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200914] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.208062] ACPI: Interpreter enabled [ 0.209282] ACPI: PM: (supports S0 S3 S4 S5) [ 0.211009] ACPI: Using IOAPIC for interrupt routing [ 0.213263] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.216629] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.231421] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.233039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.235013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.239087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.243537] acpiphp: Slot [2] registered [ 0.244134] acpiphp: Slot [5] registered [ 0.246093] acpiphp: Slot [6] registered [ 0.247068] acpiphp: Slot [7] registered [ 0.248120] acpiphp: Slot [8] registered [ 0.249104] acpiphp: Slot [9] registered [ 0.251081] acpiphp: Slot [10] registered [ 0.252070] acpiphp: Slot [3] registered [ 0.253125] acpiphp: Slot [4] registered [ 0.254066] acpiphp: Slot [11] registered [ 0.255064] acpiphp: Slot [12] registered [ 0.257096] acpiphp: Slot [13] registered [ 0.260125] acpiphp: Slot [14] registered [ 0.262088] acpiphp: Slot [15] registered [ 0.263097] acpiphp: Slot [16] registered [ 0.268116] acpiphp: Slot [17] registered [ 0.269061] acpiphp: Slot [18] registered [ 0.270058] acpiphp: Slot [19] registered [ 0.271083] acpiphp: Slot [20] registered [ 0.272062] acpiphp: Slot [21] registered [ 0.273068] acpiphp: Slot [22] registered [ 0.274058] acpiphp: Slot [23] registered [ 0.275059] acpiphp: Slot [24] registered [ 0.276080] acpiphp: Slot [25] registered [ 0.277087] acpiphp: Slot [26] registered [ 0.279097] acpiphp: Slot [27] registered [ 0.280089] acpiphp: Slot [28] registered [ 0.282086] acpiphp: Slot [29] registered [ 0.283076] acpiphp: Slot [30] registered [ 0.285081] acpiphp: Slot [31] registered [ 0.287056] PCI host bridge to bus 0000:00 [ 0.288012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.290013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.291012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.293026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.295013] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.297016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.299172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.301912] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305075] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.313011] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.317419] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.319013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.321010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.323009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.324522] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.326687] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.329029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.330696] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.334765] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.342944] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.347011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.351759] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.362012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.377016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.410045] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.423709] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.428000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.431017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.452015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.467706] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.474028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.479014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.495015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.503000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.506013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.511010] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.524012] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.532036] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.539016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.546017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.571018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.578000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.585014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.593014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.622016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.643327] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.646331] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.648278] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.650279] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.652182] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.656758] iommu: Default domain type: Passthrough [ 0.658374] SCSI subsystem initialized [ 0.659090] ACPI: bus type USB registered [ 0.660092] usbcore: registered new interface driver usbfs [ 0.662043] usbcore: registered new interface driver hub [ 0.663061] usbcore: registered new device driver usb [ 0.665131] pps_core: LinuxPPS API ver. 1 registered [ 0.666007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.668042] PTP clock support registered [ 0.670277] EDAC MC: Ver: 3.0.0 [ 0.671000] PCI: Using ACPI for IRQ routing [ 0.672793] NetLabel: Initializing [ 0.673006] NetLabel: domain hash size = 128 [ 0.675007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.676063] NetLabel: unlabeled traffic allowed by default [ 0.678231] vgaarb: loaded [ 0.679314] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.681009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.687437] clocksource: Switched to clocksource kvm-clock [ 0.798590] VFS: Disk quotas dquot_6.6.0 [ 0.800250] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.802718] *** VALIDATE ramfs *** [ 0.803909] *** VALIDATE hugetlbfs *** [ 0.805121] pnp: PnP ACPI init [ 0.808444] pnp: PnP ACPI: found 6 devices [ 0.827033] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.831163] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.832964] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.834664] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.836403] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.840328] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.842743] NET: Registered protocol family 2 [ 0.844833] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.850327] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.853284] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.859879] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.862677] TCP: Hash tables configured (established 65536 bind 65536) [ 0.864980] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.869672] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.872251] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.875427] NET: Registered protocol family 1 [ 0.877708] RPC: Registered named UNIX socket transport module. [ 0.880774] RPC: Registered udp transport module. [ 0.882126] RPC: Registered tcp transport module. [ 0.885519] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.887464] NET: Registered protocol family 44 [ 0.888801] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.890867] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.892824] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.894927] PCI: CLS 0 bytes, default 64 [ 0.897299] Unpacking initramfs... [ 2.836864] debug: unmapping init [mem 0xffff9491bcc54000-0xffff9491bffbffff] [ 2.841982] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.844992] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.848787] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.806414] Initialise system trusted keyrings [ 3.809372] Key type blacklist registered [ 3.812089] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.821789] zbud: loaded [ 3.824824] *** VALIDATE nfs *** [ 3.826057] *** VALIDATE nfs4 *** [ 3.827718] pstore: using deflate compression [ 3.831613] Platform Keyring initialized [ 3.966206] NET: Registered protocol family 38 [ 3.968206] Key type asymmetric registered [ 3.970122] Asymmetric key parser 'x509' registered [ 3.972247] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.975718] io scheduler mq-deadline registered [ 3.977572] io scheduler kyber registered [ 3.979770] io scheduler bfq registered [ 3.981787] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.985243] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.991622] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.994589] ACPI: Power Button [PWRF] [ 4.000319] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.008740] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.077403] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.085465] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.117345] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.166102] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.328699] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.333194] Non-volatile memory driver v1.3 [ 4.334438] Linux agpgart interface v0.103 [ 4.360176] virtio_blk virtio1: [vda] 132376 512-byte logical blocks (67.8 MB/64.6 MiB) [ 4.362944] vda: detected capacity change from 0 to 67776512 [ 4.382450] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.385508] vdb: detected capacity change from 0 to 1073741824 [ 4.485100] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.487200] vdc: detected capacity change from 0 to 2621440000 [ 4.520818] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.523941] vdd: detected capacity change from 0 to 2621440000 [ 4.545482] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.548820] vde: detected capacity change from 0 to 4294967296 [ 4.574988] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.578242] vdf: detected capacity change from 0 to 4294967296 [ 4.592375] libphy: Fixed MDIO Bus: probed [ 4.602158] usbcore: registered new interface driver usbserial_generic [ 4.603957] usbserial: USB Serial support registered for generic [ 4.605755] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.609852] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.612029] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.615890] mousedev: PS/2 mouse device common for all mice [ 4.620432] rtc_cmos 00:05: RTC can wake from S4 [ 4.622834] rtc_cmos 00:05: registered as rtc0 [ 4.624299] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.624430] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.628142] intel_pstate: CPU model not supported [ 4.630935] hid: raw HID events driver (C) Jiri Kosina [ 4.631739] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.632608] usbcore: registered new interface driver usbhid [ 4.635838] usbhid: USB HID core driver [ 4.637109] drop_monitor: Initializing network drop monitor service [ 4.639146] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.639201] Initializing XFRM netlink socket [ 4.642619] NET: Registered protocol family 10 [ 4.644975] Segment Routing with IPv6 [ 4.646114] NET: Registered protocol family 17 [ 4.647451] mpls_gso: MPLS GSO support [ 4.655511] RAS: Correctable Errors collector initialized. [ 4.656948] AVX version of gcm_enc/dec engaged. [ 4.658009] AES CTR mode by8 optimization enabled [ 4.731565] sched_clock: Marking stable (4731548062, 0)->(6085214967, -1353666905) [ 4.734506] registered taskstats version 1 [ 4.736970] Loading compiled-in X.509 certificates [ 4.739268] zswap: loaded using pool lzo/zbud [ 4.760399] Key type big_key registered [ 4.771712] Key type encrypted registered [ 4.772782] ima: No TPM chip found, activating TPM-bypass! [ 4.774083] ima: Allocated hash algorithm: sha1 [ 4.774956] ima: No architecture policies found [ 4.775981] evm: Initialising EVM extended attributes: [ 4.777081] evm: security.selinux [ 4.777691] evm: security.ima [ 4.778399] evm: security.capability [ 4.779193] evm: HMAC attrs: 0x1 [ 4.781130] rtc_cmos 00:05: setting system clock to 2026-04-02 21:50:30 UTC (1775166630) [ 4.786541] debug: unmapping init [mem 0xffffffffa3203000-0xffffffffa33fffff] [ 4.788533] debug: unmapping init [mem 0xffffffffa1f82000-0xffffffffa2258fff] [ 4.795106] Write protecting the kernel read-only data: 28672k [ 4.798396] debug: unmapping init [mem 0xffffffffa0603000-0xffffffffa07fffff] [ 4.799905] debug: unmapping init [mem 0xffffffffa0f14000-0xffffffffa0ffffff] [ 4.845313] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.853321] systemd[1]: Detected virtualization kvm. [ 4.854775] systemd[1]: Detected architecture x86-64. [ 4.856814] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.881161] systemd[1]: No hostname configured. [ 4.882454] systemd[1]: Set hostname to . [ 4.883932] random: systemd: uninitialized urandom read (16 bytes read) [ 4.886453] systemd[1]: Initializing machine ID from random generator. [ 4.938430] random: ln: uninitialized urandom read (6 bytes read) [ 5.023151] random: systemd: uninitialized urandom read (16 bytes read) [ 5.025609] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.029514] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.041347] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.436447] device-mapper: uevent: version 1.0.3 [ 6.440720] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.626653] random: fast init done [ 7.651475] virtio_net virtio0 ens2: renamed from eth0 [ 8.529258] scsi host0: ata_piix [ 8.642319] scsi host1: ata_piix [ 8.644640] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.651743] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.287295] random: crng init done [ 13.290857] random: 7 urandom warning(s) missed due to ratelimiting [ 14.086527] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.313514] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.564166] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.384181] SELinux: Disabled at runtime. [ 18.479915] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.492807] systemd[1]: Detected virtualization kvm. [ 18.497438] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.649584] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.653174] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.683174] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.693688] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.698612] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.707074] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.720385] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ 19.778464] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 21.066722] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.956257] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.122713] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.517980] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.594211] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 27.469991] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 28.407340] NFS: Registering the id_resolver key type [ 28.409624] Key type id_resolver registered [ 28.421149] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg259-server login: [ 56.524359] libcfs: loading out-of-tree module taints kernel. [ 56.538555] Key type ._llcrypt registered [ 56.539758] Key type .llcrypt registered [ 56.578864] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 64.079471] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 64.583455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.589039] alg: No test for adler32 (adler32-zlib) [ 65.537801] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 65.780979] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 67.383136] Key type lgssc registered [ 67.950857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.759534] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 90.558230] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 95.507265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 95.517138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 96.611534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 96.629722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 96.669427] Lustre: lustre-MDT0000: new disk, initializing [ 96.715745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 96.726137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 98.092818] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.542669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 103.593949] Lustre: 6482:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 103.608791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 103.610891] Lustre: Skipped 1 previous similar message [ 103.655571] Lustre: lustre-MDT0001: new disk, initializing [ 103.684600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 103.696450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 103.704725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 105.065561] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.347360] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 110.575657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 110.676587] Lustre: lustre-OST0000: new disk, initializing [ 110.678510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 110.695312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 112.659428] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 117.737601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 117.739884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 117.757773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 117.976672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 118.013073] Lustre: lustre-OST0001: new disk, initializing [ 118.014821] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 118.029937] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 119.863498] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 125.936523] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 127.352944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 127.357224] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 127.374758] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 130.706197] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 137.866950] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing check_logdir /tmp/testlogs/ [ 139.885542] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing yml_node [ 141.790847] Lustre: DEBUG MARKER: Client: 2.17.51.23 [ 142.882203] Lustre: DEBUG MARKER: MDS: 2.17.51.23 [ 143.819082] Lustre: DEBUG MARKER: OSS: 2.17.51.23 [ 144.411270] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Apr 2 17:52:50 EDT 2026 [ 150.590869] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 151.122510] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 163.295643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 163.296763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 163.303175] Lustre: Skipped 3 previous similar messages [ 163.312666] Lustre: Skipped 2 previous similar messages [ 168.415783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 168.419434] Lustre: Skipped 4 previous similar messages [ 169.176715] Lustre: server umount lustre-MDT0000 complete [ 172.155425] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775166797 with bad export cookie 8652702267682172242 [ 172.162338] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 172.166146] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 173.542259] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 173.546329] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 173.550326] Lustre: Skipped 1 previous similar message [ 173.556678] Lustre: Skipped 1 previous similar message [ 177.568461] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 177.579223] Lustre: Skipped 1 previous similar message [ 178.409140] Lustre: server umount lustre-MDT0001 complete [ 181.864469] Lustre: server umount lustre-OST0000 complete [ 184.700803] Lustre: server umount lustre-OST0001 complete [ 190.384568] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 196.711169] hrtimer: interrupt took 3301304 ns [ 199.561398] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 230.413447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.607760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 230.624711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 230.680463] Lustre: lustre-MDT0000: new disk, initializing [ 230.755721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.781778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 233.618325] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 240.915346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 241.021449] Lustre: 14942:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 241.059244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 241.061500] Lustre: Skipped 1 previous similar message [ 241.103546] Lustre: lustre-MDT0001: new disk, initializing [ 241.175940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.207246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 241.215317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 243.785728] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 249.173403] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 251.457446] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 255.372745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.839609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 256.891228] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 259.441308] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 264.376596] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 266.908836] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 267.074665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 269.553478] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 269.688804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 272.353614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 272.361581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 272.364201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 272.378199] Lustre: Skipped 1 previous similar message [ 272.388804] Lustre: Skipped 1 previous similar message [ 277.085379] Lustre: server umount lustre-OST0000 complete [ 282.080067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 282.086465] LustreError: Skipped 1 previous similar message [ 282.092379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 282.106661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 285.344369] Lustre: server umount lustre-MDT0000 complete [ 287.505782] LustreError: 14933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775166913 with bad export cookie 8652702267682173677 [ 287.511265] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 287.518739] LustreError: 14933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 287.691197] Lustre: server umount lustre-MDT0001 complete [ 291.264711] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 17:55:16 (1775166916) [ 303.529622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 303.834418] LustreError: 19405:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.853516] LustreError: 19405:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 303.874125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 303.877614] Lustre: Skipped 1 previous similar message [ 305.961977] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 309.217238] LustreError: 19406:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.733305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 313.479080] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 316.645626] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 318.878180] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 320.231523] Lustre: Failing over lustre-MDT0000 [ 320.336693] Lustre: server umount lustre-MDT0000 complete [ 321.248716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.251570] LustreError: 19402:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.264256] Lustre: Skipped 2 previous similar messages [ 321.294625] LustreError: 19402:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 326.628875] LustreError: 19405:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.637892] LustreError: 19405:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 331.744412] LustreError: 20115:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.755354] LustreError: 20115:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 334.804686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 334.878985] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 336.033236] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 336.037340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 336.045054] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 336.972522] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 340.920136] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 343.385711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.387906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 343.393620] Lustre: Skipped 1 previous similar message [ 343.400230] Lustre: Skipped 2 previous similar messages [ 349.638658] Lustre: server umount lustre-MDT0000 complete [ 352.012896] LustreError: 19387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775166977 with bad export cookie 8652702267682175091 [ 352.013910] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 352.019466] LustreError: 19387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 352.227623] Lustre: server umount lustre-MDT0001 complete [ 357.925061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 358.214672] LustreError: 22401:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 358.264438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 358.269768] Lustre: Skipped 2 previous similar messages [ 360.848754] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 365.695481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 368.321129] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 371.588705] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 373.955530] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 375.365444] Lustre: Failing over lustre-MDT0000 [ 375.487927] Lustre: server umount lustre-MDT0000 complete [ 376.288497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 376.292649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.298066] LustreError: 22401:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.306837] LustreError: 22401:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 390.430516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.532761] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 392.536452] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 392.737679] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 392.739348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 392.745348] Lustre: Skipped 1 previous similar message [ 392.755092] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 397.010817] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 402.916517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 402.928025] Lustre: Skipped 3 previous similar messages [ 402.931519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 414.175135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 414.288765] Lustre: server umount lustre-MDT0000 complete [ 416.778802] LustreError: 22382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775167042 with bad export cookie 8652702267682176029 [ 416.781750] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 416.784372] LustreError: 22382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 417.006948] Lustre: server umount lustre-MDT0001 complete [ 422.888387] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 17:57:28 (1775167048) [ 433.254543] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 438.976303] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 462.915202] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 469.823739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 469.951734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 469.970573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 469.973268] Lustre: Skipped 1 previous similar message [ 470.011181] Lustre: lustre-MDT0000: new disk, initializing [ 470.013510] Lustre: Skipped 1 previous similar message [ 470.078402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.082857] Lustre: Skipped 2 previous similar messages [ 470.091980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 470.096651] Lustre: Skipped 1 previous similar message [ 472.464826] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 479.004026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.081777] Lustre: 28374:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 479.242388] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 481.330704] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 486.037563] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 488.045590] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 491.593423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 492.820755] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 492.853577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 492.862071] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 494.332879] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 498.575914] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 502.895861] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 513.514530] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 513.517150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.519084] LustreError: Skipped 1 previous similar message [ 513.520664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 513.526904] Lustre: Skipped 4 previous similar messages [ 513.528462] Lustre: Skipped 4 previous similar messages [ 517.566821] Lustre: server umount lustre-OST0000 complete [ 526.192034] Lustre: server umount lustre-MDT0000 complete [ 528.134204] LustreError: 30115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775167153 with bad export cookie 8652702267682176435 [ 528.137141] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.138784] LustreError: 30115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 528.352618] Lustre: server umount lustre-MDT0001 complete [ 533.566153] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 534.926742] Key type lgssc unregistered [ 535.092228] LNet: 31942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 535.096345] LNetError: 31942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 535.108488] LNet: Removed LNI 192.168.202.159@tcp [ 535.531147] Key type .llcrypt unregistered [ 535.532748] Key type ._llcrypt unregistered [ 545.899423] Key type ._llcrypt registered [ 545.900743] Key type .llcrypt registered [ 545.946125] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 546.326867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 546.385625] alg: No test for adler32 (adler32-zlib) [ 547.243756] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 547.332824] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 548.919184] Key type lgssc registered [ 549.326861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 552.046414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 552.053421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.239042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.820100] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 558.062536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.202703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 559.652379] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 561.825064] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 563.254928] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 565.756566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 565.904985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.975255] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 568.141397] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 570.211774] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 585.818596] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:00:11 (1775167211) [ 586.511064] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 587.196079] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 589.395073] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:00:15 (1775167215) [ 592.284438] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:00:18 (1775167218) [ 592.646613] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 595.476828] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 595.479580] Lustre: Skipped 2 previous similar messages [ 596.069698] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:00:22 (1775167222) [ 602.985342] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 602.987978] Lustre: Skipped 10 previous similar messages [ 603.967163] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:00:30 (1775167230) [ 604.942436] Lustre: *** cfs_fail_loc=131b, val=2*** [ 606.208181] Lustre: *** cfs_fail_loc=131b, val=2*** [ 608.968134] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:00:35 (1775167235) [ 611.132671] Lustre: Disabling parameter general.jobid_name=TESTNAME1 in log params [ 611.134647] Lustre: Skipped 6 previous similar messages [ 613.955960] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:00:40 (1775167240) [ 619.489227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 619.496094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.500812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 619.502573] Lustre: Skipped 1 previous similar message [ 624.212157] Lustre: server umount lustre-OST0000 complete [ 629.728539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.729385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 629.733533] Lustre: Skipped 2 previous similar messages [ 629.738359] Lustre: Skipped 1 previous similar message [ 631.762655] Lustre: server umount lustre-MDT0000 complete [ 633.079263] LustreError: 33295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775167258 with bad export cookie 16207702361185162624 [ 633.082892] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.083179] LustreError: 33295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 633.204669] Lustre: server umount lustre-MDT0001 complete [ 637.079451] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 638.177767] Key type lgssc unregistered [ 638.311394] LNet: 40815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.314617] LNetError: 40815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 638.323343] LNet: Removed LNI 192.168.202.159@tcp [ 638.658234] Key type .llcrypt unregistered [ 638.659854] Key type ._llcrypt unregistered [ 645.396776] Key type ._llcrypt registered [ 645.398310] Key type .llcrypt registered [ 645.441889] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 652.107719] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 652.507672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 652.524595] alg: No test for adler32 (adler32-zlib) [ 653.385606] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 653.478840] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 655.079159] Key type lgssc registered [ 655.497443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 668.657356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 668.666960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.765910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 669.778273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 669.813529] Lustre: lustre-MDT0000: new disk, initializing [ 669.841259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.848720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 671.288357] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 675.723037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 675.765692] Lustre: 43918:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 675.787045] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 675.789255] Lustre: Skipped 1 previous similar message [ 675.824234] Lustre: lustre-MDT0001: new disk, initializing [ 675.855792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 675.872273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 675.877143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 677.281920] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 680.641599] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 681.949461] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 684.285727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 684.385854] Lustre: lustre-OST0000: new disk, initializing [ 684.388539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 684.416114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 685.118873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 685.123772] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 685.168312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 686.506049] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 689.755303] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 691.215676] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 691.296792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 692.883273] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 692.969351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 695.263869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 695.266212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 695.271421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 695.776138] Lustre: server umount lustre-OST0000 complete [ 700.897121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 700.898984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 700.901865] Lustre: Skipped 1 previous similar message [ 700.906578] Lustre: Skipped 1 previous similar message [ 706.016566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.022156] Lustre: Skipped 1 previous similar message [ 708.104369] Lustre: server umount lustre-MDT0000 complete [ 709.654728] LustreError: 44696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775167335 with bad export cookie 16884481224667421719 [ 709.658270] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 709.662603] LustreError: 44696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 709.783234] Lustre: server umount lustre-MDT0001 complete [ 713.907252] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:02:20 (1775167340) [ 716.701239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.913743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 718.597829] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 722.217395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.383205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 724.005715] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 726.143265] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 727.530855] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 730.349051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 730.497333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 732.796501] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 734.982383] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 747.703543] LustreError: 50175:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 748.085256] Lustre: Modifying parameter general.timeout=1 in log params [ 749.167681] Lustre: Modifying parameter general.timeout=4 in log params [ 749.170180] Lustre: Skipped 2 previous similar messages [ 751.248119] Lustre: Modifying parameter general.timeout=10 in log params [ 751.250263] Lustre: Skipped 5 previous similar messages [ 755.276553] Lustre: Modifying parameter general.timeout=22 in log params [ 755.278794] Lustre: Skipped 11 previous similar messages [ 763.434809] Lustre: Modifying parameter general.timeout=47 in log params [ 763.437323] Lustre: Skipped 24 previous similar messages [ 779.493554] Lustre: Modifying parameter general.timeout=95 in log params [ 779.495628] Lustre: Skipped 47 previous similar messages [ 794.351676] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:03:40 (1775167420) [ 795.396730] Lustre: 57075:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 796.759249] Lustre: 57272:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 796.764239] Lustre: 57272:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 798.797325] Lustre: 57584:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 798.801644] Lustre: 57584:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 808.096682] Lustre: 58933:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 808.100887] LustreError: 58933:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 809.507557] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 18:03:55 (1775167435) [ 814.560807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 814.563468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.568975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 818.145612] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.429995] Lustre: server umount lustre-MDT0000 complete [ 820.747451] LustreError: 47944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775167446 with bad export cookie 16884481224667422587 [ 820.749720] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 820.752512] LustreError: 47944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 820.879095] Lustre: server umount lustre-MDT0001 complete [ 831.456480] Lustre: server umount lustre-OST0000 complete [ 840.613897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.716985] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 840.730047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 840.734234] Lustre: Skipped 57 previous similar messages [ 840.831920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.178899] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 845.056099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 845.093520] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 845.103548] Lustre: 61455:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 846.597992] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 848.502410] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 849.748342] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 852.104090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 852.180708] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 854.356763] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 856.293170] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 862.268354] Lustre: 63681:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 865.249733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.250626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 865.254343] Lustre: Skipped 2 previous similar messages [ 865.256578] Lustre: Skipped 3 previous similar messages [ 865.258490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 866.199262] Lustre: server umount lustre-OST0000 complete [ 869.659595] Lustre: server umount lustre-MDT0000 complete [ 870.369214] LustreError: 61465:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 870.375567] LustreError: 61465:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 870.975890] LustreError: 61491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775167496 with bad export cookie 16884481224667424925 [ 870.976701] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 870.981272] LustreError: 61491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 871.100882] Lustre: server umount lustre-MDT0001 complete [ 874.817899] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 875.877209] Key type lgssc unregistered [ 876.008387] LNet: 64867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.010855] LNetError: 64867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.023371] LNet: Removed LNI 192.168.202.159@tcp [ 876.342950] Key type .llcrypt unregistered [ 876.344329] Key type ._llcrypt unregistered [ 885.681815] Key type ._llcrypt registered [ 885.682962] Key type .llcrypt registered [ 885.719584] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 886.013906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 886.024053] alg: No test for adler32 (adler32-zlib) [ 886.881383] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 886.965923] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 888.551126] Key type lgssc registered [ 888.888332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 891.352317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 891.358652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.521548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 893.821869] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 896.625537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.827694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 898.142416] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 899.903912] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 901.025725] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 903.156486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 903.338161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.182826] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 906.985339] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 920.496364] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 923.104577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.105597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 923.109959] Lustre: Skipped 2 previous similar messages [ 923.111949] Lustre: Skipped 1 previous similar message [ 928.224197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 928.226189] Lustre: Skipped 3 previous similar messages [ 929.043534] Lustre: server umount lustre-MDT0000 complete [ 930.275148] LustreError: 66221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775167555 with bad export cookie 7812164435880435924 [ 930.277511] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 930.279234] LustreError: 66221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 930.282921] LustreError: 66232:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 930.387442] Lustre: server umount lustre-MDT0001 complete [ 941.920603] Lustre: server umount lustre-OST0000 complete [ 951.460081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 951.558281] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 951.570227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 951.659781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 952.991571] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 955.891062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.933424] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 955.944623] Lustre: 70794:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 956.121324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 957.476823] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 959.345481] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 960.576886] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 962.884603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 962.953306] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 965.001817] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 966.883944] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 972.858783] Lustre: 73029:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 973.499501] Lustre: Modifying parameter general.jobid_var=disable in log params [ 977.889318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 977.890103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.892221] LustreError: Skipped 1 previous similar message [ 977.892855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 977.896783] Lustre: Skipped 1 previous similar message [ 977.901977] Lustre: Skipped 1 previous similar message [ 980.879850] Lustre: server umount lustre-OST0000 complete [ 983.008673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.009388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 983.012529] Lustre: Skipped 1 previous similar message [ 988.128220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 988.130727] Lustre: Skipped 2 previous similar messages [ 988.248094] Lustre: server umount lustre-MDT0000 complete [ 989.510234] LustreError: 70785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775167615 with bad export cookie 7812164435880436715 [ 989.513220] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 989.514212] LustreError: 70785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 989.621862] Lustre: server umount lustre-MDT0001 complete [ 993.097783] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 994.132461] Key type lgssc unregistered [ 994.269440] LNet: 74223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.272739] LNetError: 74223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.282315] LNet: Removed LNI 192.168.202.159@tcp [ 994.565643] Key type .llcrypt unregistered [ 994.567227] Key type ._llcrypt unregistered [ 1003.668805] Key type ._llcrypt registered [ 1003.670067] Key type .llcrypt registered [ 1003.708384] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1004.031894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1004.072084] alg: No test for adler32 (adler32-zlib) [ 1004.924882] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 1005.012626] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1006.599167] Key type lgssc registered [ 1006.989962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1009.624561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1009.632384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1010.800926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.078879] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1014.914470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.114683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1016.433586] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1018.254422] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1019.422729] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1021.626219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1021.813082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.660569] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1025.396582] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1043.183333] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1043.792032] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1043.794496] Lustre: Skipped 1 previous similar message [ 1044.946538] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1044.949896] Lustre: Skipped 3 previous similar messages [ 1046.956228] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1046.958894] Lustre: Skipped 6 previous similar messages [ 1050.959501] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1050.961666] Lustre: Skipped 12 previous similar messages [ 1059.243392] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1059.246351] Lustre: Skipped 25 previous similar messages [ 1076.527961] Lustre: 83343:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1076.829145] Lustre: Disabling parameter general.jobid_var= in log params [ 1076.831634] Lustre: Skipped 47 previous similar messages [ 1077.380812] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 18:08:23 (1775167703) [ 1077.909911] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1078.487196] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 18:08:24 (1775167704) [ 1081.311904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1081.315836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.322141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1083.361848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1083.363155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1083.363256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1083.374053] Lustre: Skipped 1 previous similar message [ 1085.516690] Lustre: server umount lustre-OST0000 complete [ 1086.943771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1086.947554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.953619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1093.600096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1093.602553] Lustre: Skipped 3 previous similar messages [ 1098.719956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1098.722908] Lustre: Skipped 1 previous similar message [ 1101.279134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1101.343718] Lustre: server umount lustre-MDT0000 complete [ 1102.550200] LustreError: 77220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775167728 with bad export cookie 3993334935334280340 [ 1102.553350] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1102.554140] LustreError: 77220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1102.672284] Lustre: server umount lustre-MDT0001 complete [ 1106.325410] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1107.511365] Key type lgssc unregistered [ 1107.641334] LNet: 84577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.644325] LNetError: 84577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1107.653370] LNet: Removed LNI 192.168.202.159@tcp [ 1107.982116] Key type .llcrypt unregistered [ 1107.983592] Key type ._llcrypt unregistered [ 1109.353462] Key type ._llcrypt registered [ 1109.354794] Key type .llcrypt registered [ 1109.391949] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1110.763412] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules [ 1111.079957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1111.087372] alg: No test for adler32 (adler32-zlib) [ 1111.944101] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 1112.030247] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1113.615175] Key type lgssc registered [ 1114.025066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1114.456189] LustreError: 85700:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 1122.068110] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1125.117181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1135.693145] LustreError: 85700:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1135.702776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1136.887658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1138.151963] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1140.925208] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 18:09:27 (1775167767) [ 1142.427348] LustreError: 86738:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1142.431071] LustreError: 86738:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1142.434110] LustreError: 86738:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1142.542636] Lustre: server umount lustre-MDT0000 complete [ 1146.526301] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1147.476621] Key type lgssc unregistered [ 1147.603329] LNet: 88195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1147.606127] LNetError: 88195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1147.615287] LNet: Removed LNI 192.168.202.159@tcp [ 1147.876176] Key type .llcrypt unregistered [ 1147.877035] Key type ._llcrypt unregistered [ 1154.497619] Key type ._llcrypt registered [ 1154.499221] Key type .llcrypt registered [ 1154.535568] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1154.900660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1154.936908] alg: No test for adler32 (adler32-zlib) [ 1155.783530] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 1155.863337] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1157.439109] Key type lgssc registered [ 1157.746512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1160.143476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1160.150671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.311786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.576091] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1165.296695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.483552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1166.723609] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1168.403582] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1169.537276] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1171.665661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1171.861051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.626080] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1175.334111] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1196.901112] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 18:10:23 (1775167823) [ 1197.422962] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1197.959726] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 18:10:24 (1775167824) [ 1201.119654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1201.122178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.127136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1201.633726] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.634214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1201.638606] Lustre: Skipped 1 previous similar message [ 1201.643153] Lustre: Skipped 1 previous similar message [ 1205.013300] Lustre: server umount lustre-MDT0000 complete [ 1206.137938] LustreError: 89354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775167831 with bad export cookie 13339432666502242802 [ 1206.140324] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1206.141552] LustreError: 89354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1206.239570] Lustre: server umount lustre-MDT0001 complete [ 1217.717764] Lustre: server umount lustre-OST0000 complete [ 1221.562367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.711975] LustreError: 93307:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.718309] LustreError: 93307:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1221.760615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.035468] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1225.871298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.111340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1227.347968] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1229.027790] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1230.177109] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1234.453535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1234.543101] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1234.547044] LustreError: 93295:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1234.557052] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1234.560291] LustreError: 95163:0:(tgt_mount.c:2565:server_fill_super()) Unable to start targets: -98 [ 1234.563276] LustreError: 95163:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 1234.565839] LustreError: 95163:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 1234.594435] Lustre: server umount lustre-OST0000 complete [ 1234.596071] LustreError: 95163:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1237.416357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1237.504262] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1237.507295] LustreError: 93295:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1237.510736] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1237.513865] LustreError: 95625:0:(tgt_mount.c:2565:server_fill_super()) Unable to start targets: -98 [ 1237.516198] LustreError: 95625:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 1237.518283] LustreError: 95625:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 1237.543582] Lustre: server umount lustre-OST0000 complete [ 1237.545450] LustreError: 95625:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1240.608476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1240.674033] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1240.676582] Lustre: Found index 0 for lustre-OST0000, updating log [ 1240.679603] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1240.702136] Lustre: lustre-OST0000: new disk, initializing [ 1240.704380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1240.813211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.723662] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1244.016979] Lustre: Failing over lustre-OST0000 [ 1244.073187] Lustre: server umount lustre-OST0000 complete [ 1245.371560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1245.372215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1245.375118] Lustre: Skipped 1 previous similar message [ 1250.527848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1250.530413] Lustre: Skipped 2 previous similar messages [ 1251.539742] Lustre: server umount lustre-MDT0000 complete [ 1252.697035] LustreError: 95183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775167878 with bad export cookie 13339432666502245532 [ 1252.699791] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1252.700802] LustreError: 95183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1252.801630] Lustre: server umount lustre-MDT0001 complete [ 1255.332915] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 18:11:21 (1775167881) [ 1259.876886] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1263.137831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1263.262994] LustreError: 98462:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.287110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.488933] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1267.149994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.587612] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1269.509830] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1271.644526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1273.649678] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1276.343625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1276.373286] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1276.401207] Lustre: lustre-OST0001: new disk, initializing [ 1276.403028] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1278.343716] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1280.193827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1280.198124] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1280.233949] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1281.340726] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1286.924278] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1294.532203] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [ 1295.842520] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.847767] Lustre: Skipped 2 previous similar messages [ 1299.283078] Lustre: server umount lustre-MDT0001 complete [ 1299.647487] LustreError: 98458:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1302.297575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1302.330800] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1302.332595] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1302.347471] Lustre: 98452:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1302.580543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1302.583495] Lustre: Skipped 3 previous similar messages [ 1303.653544] LustreError: 102420:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1303.657584] LustreError: 102420:0:(tgt_mount.c:666:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1303.661598] LustreError: 98454:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1303.685891] LustreError: 102428:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -17 [ 1303.688952] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1303.688952] [ 1303.788432] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1306.798362] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 18:12:13 (1775167933) [ 1307.622424] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1312.216599] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1317.914254] Lustre: 104293:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1325.707479] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1326.860476] Lustre: 105424:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1379.296519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1379.296819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1379.297129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1379.297136] Lustre: Skipped 3 previous similar messages [ 1379.300393] Lustre: Skipped 2 previous similar messages [ 1382.745530] Lustre: server umount lustre-MDT0000 complete [ 1383.905477] LustreError: 100725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775168009 with bad export cookie 13339432666502246358 [ 1383.907090] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1383.909523] LustreError: 100725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1384.016585] Lustre: server umount lustre-MDT0001 complete [ 1394.438567] Lustre: server umount lustre-OST0000 complete [ 1405.783896] Lustre: server umount lustre-OST0001 complete [ 1407.506576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1410.377718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1412.904993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1415.686022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1421.089464] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1424.171672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.178253] Lustre: lustre-MDT0000: reset Object Index mappings [ 1424.312907] LustreError: 109324:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1424.317103] LustreError: 109324:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1424.335032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1425.467615] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1427.981512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.987953] Lustre: lustre-MDT0001: reset Object Index mappings [ 1429.348780] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1430.318169] Lustre: 110436:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1432.468503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1434.572228] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1434.786763] LustreError: 110788:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1434.790971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1437.363807] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1438.568948] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1438.569969] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1438.570720] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1439.472185] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1442.576722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1443.819611] Lustre: 112287:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1461.335047] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 18:14:47 (1775168087) [ 1495.014638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1495.014639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.029811] Lustre: Skipped 1 previous similar message [ 1495.036024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1495.042952] Lustre: Skipped 6 previous similar messages [ 1497.930802] Lustre: server umount lustre-MDT0000 complete [ 1499.856547] LustreError: 109305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775168125 with bad export cookie 13339432666502293083 [ 1499.857360] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1499.862719] LustreError: 109305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1500.128408] LustreError: 109319:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1500.140639] LustreError: 109319:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1506.181964] Lustre: server umount lustre-MDT0001 complete [ 1514.664118] Lustre: server umount lustre-OST0000 complete [ 1532.383634] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1532.478231] Lustre: server umount lustre-OST0001 complete [ 1535.106930] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 1539.408033] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1554.302744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1554.426589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1554.444440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1554.487468] Lustre: lustre-MDT0000: new disk, initializing [ 1554.534546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1554.537659] Lustre: Skipped 3 previous similar messages [ 1554.549893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1556.622873] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1560.098230] Lustre: server umount lustre-MDT0000 complete [ 1564.147154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.287440] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1564.298488] Lustre: 116495:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1564.304462] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1566.158585] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1568.387134] Lustre: server umount lustre-MDT0000 complete [ 1572.387447] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 18:16:38 (1775168198) [ 1573.182036] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1574.001303] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 18:16:40 (1775168200) [ 1580.969287] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1585.293530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.008835] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1590.313571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.353870] Lustre: 118509:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1590.381921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1590.384313] Lustre: Skipped 1 previous similar message [ 1590.423705] Lustre: lustre-MDT0001: new disk, initializing [ 1590.476097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1590.479966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1591.923452] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1594.387338] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1597.150352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1597.278271] Lustre: lustre-OST0000: new disk, initializing [ 1597.280848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1597.814596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1597.821962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1597.886057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1599.756372] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1604.321029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1605.646543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1605.652806] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1605.693522] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1606.691655] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1611.338774] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1612.686691] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1796.891867] Lustre: 127270:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1796.894838] Lustre: 127270:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1796.897094] Lustre: 127270:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1796.900986] Lustre: 127270:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1796.904904] Lustre: 127270:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1796.907822] Lustre: 127270:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1934.994801] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:22:40 (1775168560) [ 1949.153231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1949.156675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.162213] Lustre: Skipped 5 previous similar messages [ 1949.165566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1949.168296] Lustre: Skipped 4 previous similar messages [ 1952.915255] Lustre: server umount lustre-MDT0000 complete [ 1954.571865] LustreError: 121651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775168580 with bad export cookie 13339432666502344883 [ 1954.575252] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1954.586033] LustreError: 121651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1956.834292] LustreError: 118514:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1956.844560] LustreError: 118514:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1960.835968] Lustre: server umount lustre-MDT0001 complete [ 1977.311066] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1977.356459] Lustre: server umount lustre-OST0000 complete [ 1993.183117] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1993.258886] Lustre: server umount lustre-OST0001 complete [ 1995.664151] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 1999.109619] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2012.306514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.405579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2012.417722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2012.420607] Lustre: Skipped 1 previous similar message [ 2012.460927] Lustre: lustre-MDT0000: new disk, initializing [ 2012.462581] Lustre: Skipped 1 previous similar message [ 2012.495837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2012.499141] Lustre: Skipped 5 previous similar messages [ 2012.506065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2014.064974] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2018.735059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.784720] Lustre: 130264:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2018.896428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2020.270551] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2023.516543] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2024.811966] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2027.383717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2028.935226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2028.939365] Lustre: Skipped 1 previous similar message [ 2028.942275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2028.984599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2029.892414] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2033.093797] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2036.670991] Lustre: lustre-MDD0000: changelog on [ 2037.725957] Lustre: lustre-MDD0001: changelog on [ 2577.971200] Lustre: lustre-MDD0001: changelog off [ 2578.586494] Lustre: 130271:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2578.890550] Lustre: lustre-MDD0000: changelog off [ 2581.984632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2581.985328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.987589] LustreError: Skipped 1 previous similar message [ 2581.992631] Lustre: Skipped 5 previous similar messages [ 2581.995234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2581.997769] Lustre: Skipped 9 previous similar messages [ 2586.519690] Lustre: server umount lustre-OST0000 complete [ 2592.227071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2601.951093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2602.049429] Lustre: server umount lustre-MDT0000 complete [ 2603.251963] LustreError: 130256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775169228 with bad export cookie 13339432666503149659 [ 2603.255326] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2603.257319] LustreError: 130256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2603.374794] Lustre: server umount lustre-MDT0001 complete [ 2606.718560] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2607.687860] Key type lgssc unregistered [ 2607.805315] LNet: 140549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2607.807867] LNetError: 140549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2607.817306] LNet: Removed LNI 192.168.202.159@tcp [ 2608.110119] Key type .llcrypt unregistered [ 2608.111272] Key type ._llcrypt unregistered [ 2614.094979] Key type ._llcrypt registered [ 2614.096011] Key type .llcrypt registered [ 2614.131079] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 2620.106100] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2620.348670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2620.364433] alg: No test for adler32 (adler32-zlib) [ 2621.210053] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 2621.294879] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2622.871136] Key type lgssc registered [ 2623.241294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2634.758324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2634.767399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.853778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2635.865664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2635.904462] Lustre: lustre-MDT0000: new disk, initializing [ 2635.932803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.943307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2637.204405] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2641.112497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.147529] Lustre: 143691:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2641.167883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2641.171200] Lustre: Skipped 1 previous similar message [ 2641.207332] Lustre: lustre-MDT0001: new disk, initializing [ 2641.232497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2641.242759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2641.245446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2642.450353] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2645.337418] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2646.395550] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2648.499841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2648.589766] Lustre: lustre-OST0000: new disk, initializing [ 2648.591579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2648.616719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.381075] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2652.728771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2652.732990] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2652.759860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2653.309202] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2655.362564] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_module obdecho/obdecho [ 2655.994914] LustreError: 146306:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2655.998780] LustreError: 146306:0:(obd_config.c:843:class_setup()) setup ec failed (-22) [ 2656.565777] LustreError: 146402:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2661.855812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2661.859415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.865730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2662.880300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2662.881419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2662.885512] Lustre: Skipped 1 previous similar message [ 2665.876848] Lustre: server umount lustre-MDT0000 complete [ 2667.044097] LustreError: 144469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775169292 with bad export cookie 13143820777843486595 [ 2667.046078] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2667.050269] LustreError: 144469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2667.158396] Lustre: server umount lustre-MDT0001 complete [ 2678.685560] Lustre: server umount lustre-OST0000 complete [ 2684.991150] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 2688.105078] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2699.708343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.799334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2699.811833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2699.851158] Lustre: lustre-MDT0000: new disk, initializing [ 2699.885767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.893865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2701.245213] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2705.392789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.431350] Lustre: 150071:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2705.451827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2705.454099] Lustre: Skipped 1 previous similar message [ 2705.490988] Lustre: lustre-MDT0001: new disk, initializing [ 2705.520888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2705.533174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2705.538095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2706.764584] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2709.666222] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2710.778486] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2712.871321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2714.482358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2714.485142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2714.518291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2714.744405] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2717.742588] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2719.033335] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2719.105096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2720.395142] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2720.464301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2724.833545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2724.833645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2724.835477] LustreError: Skipped 1 previous similar message [ 2724.839945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2724.843633] Lustre: Skipped 3 previous similar messages [ 2727.179960] Lustre: server umount lustre-OST0000 complete [ 2729.952587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.952967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2729.955677] Lustre: Skipped 2 previous similar messages [ 2729.958139] Lustre: Skipped 1 previous similar message [ 2734.485155] Lustre: server umount lustre-MDT0000 complete [ 2735.661165] LustreError: 150850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775169361 with bad export cookie 13143820777843487533 [ 2735.663575] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2735.664751] LustreError: 150850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2735.778945] Lustre: server umount lustre-MDT0001 complete [ 2736.865980] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:36:03 (1775169363) [ 2737.362597] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2737.910567] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:36:04 (1775169364) [ 2740.325189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2740.459584] LustreError: 154074:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2740.466804] LustreError: 154074:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2740.479797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2740.481962] Lustre: Skipped 1 previous similar message [ 2741.765495] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2744.538391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2745.933504] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2747.640354] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2748.775549] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2750.924148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2752.781802] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2754.450168] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2775.236295] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 2775.521777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2775.521981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.528574] Lustre: Skipped 2 previous similar messages [ 2778.072280] Lustre: server umount lustre-MDT0000 complete [ 2779.267345] LustreError: 154101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775169404 with bad export cookie 13143820777843488408 [ 2779.269764] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2779.272900] LustreError: 154101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2780.350592] LustreError: 154100:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2784.671730] LustreError: 154070:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2784.671974] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2784.678331] LustreError: 154070:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2784.680594] Lustre: Skipped 5 previous similar messages [ 2785.451652] Lustre: server umount lustre-MDT0001 complete [ 2787.530669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2791.691922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.766428] LustreError: 157599:0:(llog.c:1646:llog_backup()) MGC192.168.202.159@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2791.769094] Lustre: 157599:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.159@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2800.702618] LustreError: 157603:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.822627] LustreError: 157620:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.829164] LustreError: 157620:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2817.214489] LustreError: 157603:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.214489] LustreError: 157604:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.214503] LustreError: 157603:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 8 previous similar messages [ 2817.219043] LustreError: 157604:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 8 previous similar messages [ 2836.671100] LustreError: 157605:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.676795] LustreError: 157605:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 14 previous similar messages [ 2852.447306] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2852.449995] Lustre: 157599:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.159@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2869.215805] LustreError: 157620:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.222093] LustreError: 157620:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 21 previous similar messages [ 2879.455232] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2879.457630] Lustre: 154066:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.159@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2893.286306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2893.288906] Lustre: Skipped 2 previous similar messages [ 2894.562279] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2897.426283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2898.402359] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2898.404810] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb6683b5bc6c28fce [ 2899.786632] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2901.415582] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2902.940983] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 2903.524959] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2903.526470] Lustre: Skipped 1 previous similar message [ 2903.975128] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2908.552096] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec [ 2913.762197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.763314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2913.763761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2913.768040] Lustre: Skipped 2 previous similar messages [ 2913.775658] Lustre: Skipped 2 previous similar messages [ 2914.900352] Lustre: server umount lustre-MDT0000 complete [ 2916.063052] LustreError: 154055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775169541 with bad export cookie 13143820777843494862 [ 2916.065291] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2916.067570] LustreError: 154055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2916.174502] Lustre: server umount lustre-MDT0001 complete [ 2927.647446] Lustre: server umount lustre-OST0000 complete [ 2934.416240] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 2937.652085] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2948.774361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.859272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2948.871538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2948.873804] Lustre: Skipped 1 previous similar message [ 2948.907320] Lustre: lustre-MDT0000: new disk, initializing [ 2948.908690] Lustre: Skipped 1 previous similar message [ 2948.933149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2948.935649] Lustre: Skipped 1 previous similar message [ 2948.942074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2950.109984] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2954.190720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.231492] Lustre: 162422:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2954.326025] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2955.634284] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2958.760824] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2959.890812] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2962.121134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2963.895261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2963.928254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2964.072888] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2967.073108] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2968.319313] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2968.380055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2969.540742] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2969.600494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2974.178650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2974.178807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.179601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2974.180881] LustreError: Skipped 1 previous similar message [ 2974.190281] Lustre: Skipped 2 previous similar messages [ 2976.270740] Lustre: server umount lustre-OST0000 complete [ 2983.569855] Lustre: server umount lustre-MDT0000 complete [ 2984.415986] LustreError: 162433:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.419941] LustreError: 162433:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 21 previous similar messages [ 2984.746958] LustreError: 164164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775169610 with bad export cookie 13143820777843511662 [ 2984.749898] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2984.750151] LustreError: 164164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2984.854888] Lustre: server umount lustre-MDT0001 complete [ 2987.513880] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:40:13 (1775169613) [ 2989.553463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.907194] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2993.602338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2994.970175] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2996.746175] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2997.952954] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3000.140304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3002.155078] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3003.985316] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3023.356072] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:40:49 (1775169649) [ 3024.865079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3024.865286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3024.873058] Lustre: Skipped 2 previous similar messages [ 3030.411501] Lustre: server umount lustre-OST0000 complete [ 3037.713510] Lustre: server umount lustre-MDT0000 complete [ 3038.887064] LustreError: 168794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775169664 with bad export cookie 13143820777843512530 [ 3038.889811] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3038.892263] LustreError: 168794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3038.988441] Lustre: server umount lustre-MDT0001 complete [ 3042.437021] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3043.354589] Key type lgssc unregistered [ 3043.480418] LNet: 169816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3043.483429] LNetError: 169816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3043.494309] LNet: Removed LNI 192.168.202.159@tcp [ 3043.783059] Key type .llcrypt unregistered [ 3043.784177] Key type ._llcrypt unregistered [ 3050.681976] Key type ._llcrypt registered [ 3050.683327] Key type .llcrypt registered [ 3050.720693] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3051.012502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3051.034716] alg: No test for adler32 (adler32-zlib) [ 3051.872125] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 3051.944837] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3053.519148] Key type lgssc registered [ 3053.862983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3056.217839] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3056.223310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3071.647388] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3076.767246] LustreError: 171018:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 3081.887380] LustreError: 171018:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3081.891966] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3081.894146] LustreError: MGC192.168.202.159@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3081.898236] LustreError: 171018:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3081.901766] LustreError: 171018:0:(tgt_mount.c:2565:server_fill_super()) Unable to start targets: -22 [ 3081.904985] LustreError: 171018:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 3081.924358] Lustre: server umount lustre-OST0000 complete [ 3081.925697] LustreError: 171018:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3084.564996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.701523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3086.840189] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3089.381767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3089.491449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3090.774470] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3092.437338] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3093.513239] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3095.613622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3095.686633] LustreError: 173102:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3095.690273] Lustre: 173102:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.202.159@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3095.737193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.524371] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3099.200331] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3099.895318] LustreError: lustre-MDT0000-lwp-MDT0001: operation fld_query to node 0@lo failed: rc = -107 [ 3099.898090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.903063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3104.735774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3104.738342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3104.742931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3106.006768] Lustre: server umount lustre-MDT0000 complete [ 3107.111967] LustreError: 171498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775169732 with bad export cookie 17222707831315071574 [ 3107.114646] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3107.115235] LustreError: 171498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3108.063177] LustreError: 172226:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003f941e2d x1861400319249152/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3108.106986] Lustre: server umount lustre-MDT0001 complete [ 3118.484346] Lustre: server umount lustre-OST0000 complete [ 3121.451338] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3124.622295] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3136.681814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.759417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3136.769617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3136.802683] Lustre: lustre-MDT0000: new disk, initializing [ 3136.823389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3136.828596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3137.996776] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3141.939272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.976778] Lustre: 176479:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3141.997937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3141.999589] Lustre: Skipped 1 previous similar message [ 3142.031554] Lustre: lustre-MDT0001: new disk, initializing [ 3142.060447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3142.070918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3142.073931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3143.257453] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3146.205758] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3147.352091] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3149.520542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.606490] Lustre: lustre-OST0000: new disk, initializing [ 3149.608045] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3151.415562] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3153.908581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3153.911829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3153.936050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3154.296967] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3155.559871] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3155.628197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3156.915304] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3156.983868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3159.010179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3159.010442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.010862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3159.010866] Lustre: Skipped 3 previous similar messages [ 3159.012411] LustreError: Skipped 1 previous similar message [ 3159.017273] Lustre: Skipped 2 previous similar messages [ 3163.660405] Lustre: server umount lustre-OST0000 complete [ 3167.711720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3167.713449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.716499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3167.717839] Lustre: Skipped 1 previous similar message [ 3170.962701] Lustre: server umount lustre-MDT0000 complete [ 3172.111139] LustreError: 176471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775169797 with bad export cookie 17222707831315072274 [ 3172.114182] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3172.114386] LustreError: 176471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3172.222279] Lustre: server umount lustre-MDT0001 complete [ 3175.144480] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:43:21 (1775169801) [ 3180.755467] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3181.676200] Key type lgssc unregistered [ 3181.802370] LNet: 180996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3181.804532] LNetError: 180996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3181.813320] LNet: Removed LNI 192.168.202.159@tcp [ 3182.102888] Key type .llcrypt unregistered [ 3182.104236] Key type ._llcrypt unregistered [ 3188.606182] Key type ._llcrypt registered [ 3188.607347] Key type .llcrypt registered [ 3188.641943] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3189.045649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3189.051707] alg: No test for adler32 (adler32-zlib) [ 3189.891283] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 3189.969994] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3191.551140] Key type lgssc registered [ 3191.899057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3194.319459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3194.325604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.470726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3196.667612] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3199.316457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.419097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3200.641981] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3202.347063] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3203.435229] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3205.775526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3205.837204] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3205.838905] LustreError: 183808:0:(llog.c:1687:llog_backup()) MGC192.168.202.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3205.841989] Lustre: 183808:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.202.159@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3205.872346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3207.574496] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3209.207944] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3210.848439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3210.850692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3210.852761] Lustre: Skipped 1 previous similar message [ 3215.955706] Lustre: server umount lustre-MDT0000 complete [ 3216.351885] LustreError: 182173:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.356695] LustreError: 182173:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 3217.140356] LustreError: 182889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775169842 with bad export cookie 3450795418037836608 [ 3217.144120] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3217.145474] LustreError: 182889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3219.039214] LustreError: 182883:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f8f2737f x1861400463951616/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3219.086595] Lustre: server umount lustre-MDT0001 complete [ 3230.612115] Lustre: server umount lustre-OST0000 complete [ 3233.336347] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3236.145019] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3246.879036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.962802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3246.973701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3247.008916] Lustre: lustre-MDT0000: new disk, initializing [ 3247.029733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.036570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3248.246332] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3252.111809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3252.145461] Lustre: 187183:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3252.159538] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3252.160615] Lustre: Skipped 1 previous similar message [ 3252.161661] LustreError: 187949:0:(llog.c:1687:llog_backup()) MGC192.168.202.159@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3252.163950] LustreError: 187949:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3252.165685] Lustre: 187949:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.159@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3252.171514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3252.172794] Lustre: Skipped 1 previous similar message [ 3252.202996] Lustre: lustre-MDT0001: new disk, initializing [ 3252.230771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3252.241508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3252.244754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3253.371601] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3256.243918] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3257.310626] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3259.369139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3259.457785] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3259.459796] Lustre: Skipped 1 previous similar message [ 3259.461105] LustreError: 188902:0:(llog.c:1687:llog_backup()) MGC192.168.202.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3259.463256] LustreError: 188902:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3259.465474] Lustre: 188902:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.159@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3259.467676] Lustre: 188902:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3259.475910] Lustre: lustre-OST0000: new disk, initializing [ 3259.477784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3261.258380] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3264.155537] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3264.999915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3265.002994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3265.014541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3265.396909] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3265.463661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3266.656229] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3266.721263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3270.113819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.114014] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3270.116740] Lustre: Skipped 1 previous similar message [ 3270.118885] LustreError: Skipped 1 previous similar message [ 3270.119339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3270.123186] Lustre: Skipped 3 previous similar messages [ 3273.418664] Lustre: server umount lustre-OST0000 complete [ 3275.232332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3275.233050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3275.235548] Lustre: Skipped 1 previous similar message [ 3280.351870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3280.354048] Lustre: Skipped 2 previous similar messages [ 3280.659792] Lustre: server umount lustre-MDT0000 complete [ 3281.805069] LustreError: 187963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775169907 with bad export cookie 3450795418037837308 [ 3281.807389] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3281.808383] LustreError: 187963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3281.916220] Lustre: server umount lustre-MDT0001 complete [ 3284.645131] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 18:45:10 (1775169910) [ 3289.675484] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3293.076754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.207681] LustreError: 191885:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.222237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3293.224013] Lustre: Skipped 1 previous similar message [ 3294.506163] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3297.377541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.751268] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3299.665408] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3301.703661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3303.642852] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3306.473330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3306.530597] Lustre: lustre-OST0001: new disk, initializing [ 3306.532881] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3308.426222] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3311.105662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3311.111379] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3311.133452] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3313.820949] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3318.351164] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3323.925133] Lustre: DEBUG MARKER: ADD OST3 [ 3325.491491] loop: module loaded [ 3325.509788] loop0: detected capacity change from 0 to 204800000 [ 3325.574892] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3325.577847] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3325.581607] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3327.608839] loop0: detected capacity change from 0 to 204800000 [ 3329.514270] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3329.575514] Lustre: lustre-OST0003: new disk, initializing [ 3329.577823] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3329.612350] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3329.614794] Lustre: Skipped 3 previous similar messages [ 3330.498289] Lustre: DEBUG MARKER: STOP OST3 [ 3331.015224] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3331.051414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3331.054303] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3331.064825] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3331.186620] Lustre: Failing over lustre-OST0003 [ 3331.234689] Lustre: server umount lustre-OST0003 complete [ 3331.695822] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3333.023075] LustreError: 196188:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3335.586811] Lustre: DEBUG MARKER: START OST3 again [ 3336.164214] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3336.164468] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3336.166932] LustreError: 193339:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3336.170767] Lustre: Skipped 1 previous similar message [ 3336.177344] LustreError: 193339:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3337.730336] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3337.813209] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3338.986327] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3339.150277] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3339.150489] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3339.513665] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3353.072709] Lustre: Failing over lustre-OST0003 [ 3353.108149] Lustre: server umount lustre-OST0003 complete [ 3354.592313] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3354.592446] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.598558] LustreError: Skipped 1 previous similar message [ 3354.599145] LustreError: 193338:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.603187] Lustre: Skipped 1 previous similar message [ 3354.611302] LustreError: 193338:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3363.299870] LustreError: 193338:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3363.305157] LustreError: 193338:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3368.418318] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.420400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3368.422317] Lustre: Skipped 3 previous similar messages [ 3368.424273] Lustre: Skipped 2 previous similar messages [ 3371.742754] Lustre: server umount lustre-MDT0000 complete [ 3372.966025] LustreError: 191868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775169998 with bad export cookie 3450795418037838176 [ 3372.970254] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3372.973876] LustreError: 191868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3373.535800] LustreError: 193523:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3373.541363] LustreError: 193523:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3377.568424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3377.570940] Lustre: Skipped 4 previous similar messages [ 3387.359130] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3387.439861] Lustre: server umount lustre-MDT0001 complete [ 3388.749763] Lustre: server umount lustre-OST0000 complete [ 3390.009829] Lustre: server umount lustre-OST0001 complete [ 3391.957258] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3394.369550] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3405.575193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3405.654773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3405.664056] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3405.697067] Lustre: lustre-MDT0000: new disk, initializing [ 3405.718729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3405.720544] Lustre: Skipped 1 previous similar message [ 3405.725132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3406.909542] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3410.860568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.902162] Lustre: 201208:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3410.999663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3412.248814] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3415.335664] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3416.456728] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3418.679426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3420.614419] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3420.789797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3420.824447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3423.560595] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3424.882887] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3424.960021] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3426.323957] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3426.395536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3431.394175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3431.394598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3431.395966] LustreError: Skipped 1 previous similar message [ 3431.398907] Lustre: Skipped 2 previous similar messages [ 3431.399412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3431.402667] Lustre: Skipped 3 previous similar messages [ 3433.105279] Lustre: server umount lustre-OST0000 complete [ 3436.512731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3440.471286] Lustre: server umount lustre-MDT0000 complete [ 3441.631966] LustreError: 202220:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3441.636837] LustreError: 202220:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 3441.641249] LustreError: 201200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775170067 with bad export cookie 3450795418037840430 [ 3441.644143] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3441.645305] LustreError: 201200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3441.757260] Lustre: server umount lustre-MDT0001 complete [ 3442.871361] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 18:47:49 (1775170069) [ 3447.715809] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3450.716422] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3461.917123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.998531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3462.009098] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3462.010938] Lustre: Skipped 3 previous similar messages [ 3462.046380] Lustre: lustre-MDT0000: new disk, initializing [ 3462.047745] Lustre: Skipped 2 previous similar messages [ 3462.077104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3462.080192] Lustre: Skipped 2 previous similar messages [ 3463.351092] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3467.298855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3467.333979] Lustre: 207365:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3467.431377] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3468.685226] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3471.571990] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3472.687701] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3474.808364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3474.927269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3474.929419] Lustre: Skipped 4 previous similar messages [ 3476.572065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3476.755419] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3479.736684] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3481.023189] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3481.086987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3482.346041] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3482.419729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3486.690111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3486.690652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3486.692117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3486.695163] Lustre: Skipped 4 previous similar messages [ 3486.696786] Lustre: Skipped 2 previous similar messages [ 3489.165861] Lustre: server umount lustre-OST0000 complete [ 3496.467837] Lustre: server umount lustre-MDT0000 complete [ 3496.928180] LustreError: 208160:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3496.932742] LustreError: 208160:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3497.654951] LustreError: 208143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775170123 with bad export cookie 3450795418037841312 [ 3497.657638] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3497.658238] LustreError: 208143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3497.771188] Lustre: server umount lustre-MDT0001 complete [ 3500.349278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3501.647675] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3504.514640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.858799] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3507.546769] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3508.635878] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3510.711304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3512.536577] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3514.152235] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3523.039971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3523.043553] LustreError: Skipped 1 previous similar message [ 3526.665796] Lustre: server umount lustre-OST0000 complete [ 3534.032918] Lustre: server umount lustre-MDT0000 complete [ 3535.215199] LustreError: 211154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775170160 with bad export cookie 3450795418037842194 [ 3535.218484] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3535.219693] LustreError: 211154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3535.327582] Lustre: server umount lustre-MDT0001 complete [ 3646.434203] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3647.457417] Key type lgssc unregistered [ 3647.598514] LNet: 214481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3647.601552] LNetError: 214481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3647.613438] LNet: Removed LNI 192.168.202.159@tcp [ 3647.938152] Key type .llcrypt unregistered [ 3647.940141] Key type ._llcrypt unregistered [ 3656.024777] Key type ._llcrypt registered [ 3656.026112] Key type .llcrypt registered [ 3656.063579] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3662.143860] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3662.565777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3662.603952] alg: No test for adler32 (adler32-zlib) [ 3663.461604] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 3663.547085] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3665.127155] Key type lgssc registered [ 3665.499401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3676.608967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3676.616232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.702685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3677.713729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3677.752782] Lustre: lustre-MDT0000: new disk, initializing [ 3677.775677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3677.782256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3679.039495] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3683.047244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3683.089224] Lustre: 217771:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3683.112704] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3683.114606] Lustre: Skipped 1 previous similar message [ 3683.151081] Lustre: lustre-MDT0001: new disk, initializing [ 3683.181481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3683.192740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3683.195856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3684.506389] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3687.540313] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3688.712318] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3690.961837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3691.066000] Lustre: lustre-OST0000: new disk, initializing [ 3691.068995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3691.100148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.034289] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3693.878045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3693.882513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3693.911185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3696.139352] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3697.573446] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3697.671108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3699.096921] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3699.170531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3704.289526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3704.289989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.292126] LustreError: Skipped 1 previous similar message [ 3704.292595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3704.295197] Lustre: Skipped 1 previous similar message [ 3704.300615] Lustre: Skipped 1 previous similar message [ 3709.409489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3709.411033] Lustre: Skipped 1 previous similar message [ 3714.015127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3714.070180] Lustre: server umount lustre-OST0000 complete [ 3719.135846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3719.138335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3719.142958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3721.301185] Lustre: server umount lustre-MDT0000 complete [ 3722.529391] LustreError: 218548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775170348 with bad export cookie 2979731192673512890 [ 3722.532818] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3722.533913] LustreError: 218548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3722.652335] Lustre: server umount lustre-MDT0001 complete [ 3728.375522] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3731.723399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.878047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3733.181563] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3735.952336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3736.067663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3737.380792] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3738.342089] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3740.571022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3742.634410] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3745.517160] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3745.588498] Lustre: lustre-OST0001: new disk, initializing [ 3745.590549] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3745.624639] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3745.626295] Lustre: Skipped 1 previous similar message [ 3747.584948] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3749.318537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3749.322251] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3749.356067] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3753.028720] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3759.711853] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3768.648519] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 18:53:14 (1775170394) [ 3769.825747] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3769.827987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3769.830693] Lustre: Skipped 3 previous similar messages [ 3769.832698] Lustre: Skipped 2 previous similar messages [ 3774.945566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3774.947580] Lustre: Skipped 5 previous similar messages [ 3775.454125] Lustre: server umount lustre-MDT0000 complete [ 3776.612217] LustreError: 222258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775170402 with bad export cookie 2979731192673513758 [ 3776.615258] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3776.618222] LustreError: 222258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3776.730122] Lustre: server umount lustre-MDT0001 complete [ 3788.295023] Lustre: server umount lustre-OST0000 complete [ 3799.775368] Lustre: server umount lustre-OST0001 complete [ 3801.752606] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3804.833130] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3816.614603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3816.710660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3816.721450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3816.756098] Lustre: lustre-MDT0000: new disk, initializing [ 3816.778486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3816.787262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3818.076484] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3822.100818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3822.144665] Lustre: 228422:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3822.258256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3823.665205] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3826.796871] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3827.952640] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3830.250843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3830.353483] Lustre: lustre-OST0000: new disk, initializing [ 3830.355014] Lustre: Skipped 1 previous similar message [ 3830.356849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3830.358965] Lustre: Skipped 2 previous similar messages [ 3832.314607] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3832.376357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3832.380269] Lustre: Skipped 1 previous similar message [ 3832.383032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3832.422328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3835.505333] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3836.886096] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3836.962371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3838.386967] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3838.458678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3841.168485] Lustre: server umount lustre-OST0000 complete [ 3842.528561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.529128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3842.532975] Lustre: Skipped 2 previous similar messages [ 3852.794481] Lustre: server umount lustre-MDT0000 complete [ 3854.071286] LustreError: 228414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775170479 with bad export cookie 2979731192673515886 [ 3854.074147] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3854.075742] LustreError: 228414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3863.570320] Lustre: server umount lustre-MDT0001 complete [ 3866.440270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.596795] LustreError: 232231:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3866.605879] LustreError: 232231:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3866.620949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3866.623965] Lustre: Skipped 2 previous similar messages [ 3867.944321] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3870.820396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3872.316379] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3874.173619] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3875.425171] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3877.621712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3879.609183] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3881.338760] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4026.338807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4026.339219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4026.339477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4026.339482] Lustre: Skipped 5 previous similar messages [ 4026.342340] Lustre: Skipped 1 previous similar message [ 4030.485241] Lustre: server umount lustre-OST0000 complete [ 4035.039904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4035.043047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.857330] Lustre: server umount lustre-MDT0000 complete [ 4039.155326] LustreError: 232212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775170664 with bad export cookie 2979731192673516768 [ 4039.159199] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4039.160740] LustreError: 232212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4039.298060] Lustre: server umount lustre-MDT0001 complete [ 4145.415982] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4146.401186] Key type lgssc unregistered [ 4146.541369] LNet: 235491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4146.544876] LNetError: 235491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4146.555248] LNet: Removed LNI 192.168.202.159@tcp [ 4146.856107] Key type .llcrypt unregistered [ 4146.857682] Key type ._llcrypt unregistered [ 4154.652518] Key type ._llcrypt registered [ 4154.653738] Key type .llcrypt registered [ 4154.693208] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 4160.694691] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4161.045060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4161.079563] alg: No test for adler32 (adler32-zlib) [ 4161.926644] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 4162.000851] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4163.575124] Key type lgssc registered [ 4163.910184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4177.453826] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4180.678551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4180.684302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4181.779508] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4181.793899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4181.834062] Lustre: lustre-MDT0000: new disk, initializing [ 4181.858429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4181.864494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4183.163600] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4187.209052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.248636] Lustre: 239495:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4187.269242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4187.270932] Lustre: Skipped 1 previous similar message [ 4187.304481] Lustre: lustre-MDT0001: new disk, initializing [ 4187.334604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4187.344749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4187.347747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4188.632856] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4190.871839] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4193.048810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4193.176920] Lustre: lustre-OST0000: new disk, initializing [ 4193.179932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4193.220905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4194.875944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4194.878858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4194.922535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4195.234164] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4199.463796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4199.530910] Lustre: lustre-OST0001: new disk, initializing [ 4199.532822] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4199.568317] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4201.591233] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4204.544650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4204.547825] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4204.580144] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4207.152525] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4212.963525] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4223.456239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4223.459673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4223.465664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4224.996862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4224.998243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4225.000919] Lustre: Skipped 2 previous similar messages [ 4225.003647] Lustre: Skipped 2 previous similar messages [ 4226.987881] Lustre: server umount lustre-MDT0000 complete [ 4228.240379] LustreError: 240273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775170853 with bad export cookie 13908977573504993163 [ 4228.243927] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4228.245732] LustreError: 240273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4228.373062] Lustre: server umount lustre-MDT0001 complete [ 4240.326863] Lustre: server umount lustre-OST0000 complete [ 4251.807730] Lustre: server umount lustre-OST0001 complete [ 4256.378674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4259.491057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4262.201272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4265.304028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4271.238056] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4274.520470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4274.526643] Lustre: lustre-MDT0000: reset Object Index mappings [ 4274.686986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4275.917875] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4278.673200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.679758] Lustre: lustre-MDT0001: reset Object Index mappings [ 4280.139677] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4281.104790] Lustre: 248035:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4283.295924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4283.417761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.420785] Lustre: Skipped 1 previous similar message [ 4284.451792] LustreError: 248389:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4284.454383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4284.459256] LustreError: 248389:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4285.323248] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4288.200813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4289.326227] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4290.122729] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4293.216587] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4294.352102] Lustre: 249875:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4304.351808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4304.354045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4304.357792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4304.359582] Lustre: Skipped 1 previous similar message [ 4306.657307] Lustre: server umount lustre-MDT0000 complete [ 4307.901081] LustreError: 246908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775170933 with bad export cookie 13908977573504999939 [ 4307.904663] LustreError: 246908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4307.905353] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4308.019290] Lustre: server umount lustre-MDT0001 complete [ 4320.197564] Lustre: server umount lustre-OST0000 complete [ 4331.749463] Lustre: server umount lustre-OST0001 complete [ 4338.402821] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:02:44 (1775170964) [ 4343.895027] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 4347.324838] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4358.889310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4358.977025] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4358.988236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4359.029768] Lustre: lustre-MDT0000: new disk, initializing [ 4359.054131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4359.056199] Lustre: Skipped 1 previous similar message [ 4359.061979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4360.402230] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4364.497310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.540410] Lustre: 254090:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4364.650296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4365.999458] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4369.004643] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4370.236754] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4372.424153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4372.534518] Lustre: lustre-OST0000: new disk, initializing [ 4372.536074] Lustre: Skipped 1 previous similar message [ 4372.538222] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4372.540434] Lustre: Skipped 2 previous similar messages [ 4374.365187] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4374.523662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4374.526442] Lustre: Skipped 1 previous similar message [ 4374.529044] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4374.564040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4377.529681] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4378.916322] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4378.994201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4380.296412] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4380.365635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4384.737951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4384.738490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.742956] LustreError: Skipped 1 previous similar message [ 4384.746101] Lustre: Skipped 3 previous similar messages [ 4384.746614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4384.753183] Lustre: Skipped 4 previous similar messages [ 4387.094082] Lustre: server umount lustre-OST0000 complete [ 4389.857348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.858276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4389.862062] Lustre: Skipped 2 previous similar messages [ 4394.524623] Lustre: server umount lustre-MDT0000 complete [ 4394.976067] LustreError: 254875:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.980758] LustreError: 254875:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4395.731382] LustreError: 254083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775171021 with bad export cookie 13908977573505002032 [ 4395.734502] LustreError: 254083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4395.734538] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4395.853083] Lustre: server umount lustre-MDT0001 complete [ 4401.567657] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4404.915939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4405.059733] LustreError: 258596:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4405.075291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4405.077146] Lustre: Skipped 2 previous similar messages [ 4406.329866] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4409.088632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4410.479066] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4411.402097] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4413.523851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4415.503379] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4418.249614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4418.311930] Lustre: lustre-OST0001: new disk, initializing [ 4418.313804] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4419.398267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4419.401049] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4419.433754] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4420.132972] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4424.484182] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4425.661549] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4432.250897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4432.255498] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4434.914252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4434.914517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4434.916690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4434.916698] Lustre: Skipped 1 previous similar message [ 4434.917778] Lustre: Skipped 1 previous similar message [ 4439.963269] Lustre: server umount lustre-MDT0000 complete [ 4440.033352] LustreError: 258597:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4440.037037] LustreError: 258597:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4441.191181] LustreError: 258578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775171066 with bad export cookie 13908977573505002914 [ 4441.194218] LustreError: 258578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4441.194625] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4441.303442] Lustre: server umount lustre-MDT0001 complete [ 4452.350771] Lustre: server umount lustre-OST0000 complete [ 4463.842732] Lustre: server umount lustre-OST0001 complete [ 4468.944091] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4472.238039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4472.378763] LustreError: 263872:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4472.397114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4472.400189] Lustre: Skipped 3 previous similar messages [ 4473.679111] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4476.455168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4477.849235] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4478.790631] Lustre: 264978:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4480.991763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4483.109364] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4483.172959] LustreError: 265332:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4483.180240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4485.921466] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4487.020755] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4487.880380] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4490.936680] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4492.072403] Lustre: 266825:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4498.676984] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4498.679847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4501.564091] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:05:27 (1775171127) [ 4515.166772] Lustre: server umount lustre-MDT0000 complete [ 4516.337106] LustreError: 263852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775171142 with bad export cookie 13908977573505009207 [ 4516.341935] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4517.856712] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4517.857752] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4517.858069] LustreError: 263867:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.861695] Lustre: Skipped 5 previous similar messages [ 4517.863440] Lustre: Skipped 4 previous similar messages [ 4522.538716] Lustre: server umount lustre-MDT0001 complete [ 4538.335088] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4538.371925] Lustre: server umount lustre-OST0000 complete [ 4545.618887] Lustre: server umount lustre-OST0001 complete [ 4547.690960] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 4550.897153] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4565.276357] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4568.685065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4568.769832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4568.779338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4568.815236] Lustre: lustre-MDT0000: new disk, initializing [ 4568.842595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4570.124100] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4574.235325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4574.275380] Lustre: 271006:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4574.375862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4575.658506] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4580.153562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4581.225540] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4582.169755] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4586.250181] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4587.746146] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4588.106809] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4592.377328] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4602.250820] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 19:07:08 (1775171228) [ 4603.362776] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4603.364657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4603.366228] Lustre: Skipped 3 previous similar messages [ 4603.368119] Lustre: Skipped 4 previous similar messages [ 4609.117665] Lustre: server umount lustre-MDT0000 complete [ 4610.254280] LustreError: 271000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775171235 with bad export cookie 13908977573505017075 [ 4610.257539] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4610.257780] LustreError: 271000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4610.375462] Lustre: server umount lustre-MDT0001 complete [ 4621.828457] Lustre: server umount lustre-OST0000 complete [ 4632.341079] Lustre: server umount lustre-OST0001 complete [ 4638.233883] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 4641.400830] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4653.677370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4653.764284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4653.766990] Lustre: Skipped 2 previous similar messages [ 4653.774993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4653.777619] Lustre: Skipped 4 previous similar messages [ 4653.816123] Lustre: lustre-MDT0000: new disk, initializing [ 4653.818131] Lustre: Skipped 3 previous similar messages [ 4653.843365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4653.845178] Lustre: Skipped 7 previous similar messages [ 4653.850731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4653.853178] Lustre: Skipped 3 previous similar messages [ 4655.108178] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4659.288814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4659.329517] Lustre: 278041:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4659.432982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4659.436583] Lustre: Skipped 2 previous similar messages [ 4660.706538] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4663.714863] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4664.893601] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4667.116890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4668.764193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4669.082378] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4672.083075] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4672.779696] Lustre: Mounted lustre-client [ 4673.085047] LustreError: 280539:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff949101abb800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4673.115172] Lustre: Unmounted lustre-client [ 4678.114233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4678.115304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4678.119206] Lustre: Skipped 1 previous similar message [ 4678.123095] Lustre: Skipped 5 previous similar messages [ 4681.997885] Lustre: server umount lustre-OST0000 complete [ 4685.279839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4689.433355] Lustre: server umount lustre-MDT0000 complete [ 4690.628082] LustreError: 278034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775171316 with bad export cookie 13908977573505018601 [ 4690.631565] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4690.633269] LustreError: 278034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4690.748763] Lustre: server umount lustre-MDT0001 complete [ 4694.132118] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4695.139853] Key type lgssc unregistered [ 4728.031493] LNet: 281818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4728.035356] LNetError: 281818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4801.765320] LNet: Removed LNI 192.168.202.159@tcp [ 4802.061087] Key type .llcrypt unregistered [ 4802.062672] Key type ._llcrypt unregistered [ 4807.684560] Key type ._llcrypt registered [ 4807.685550] Key type .llcrypt registered [ 4807.719307] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 4815.394595] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4815.741791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4815.750937] alg: No test for adler32 (adler32-zlib) [ 4816.618197] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 4816.702551] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4818.279130] Key type lgssc registered [ 4818.721288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4829.241276] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:10:55 (1775171455) [ 4831.781172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4831.786635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.907795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4832.920973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4832.974197] Lustre: lustre-MDT0000: new disk, initializing [ 4833.002209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4833.008705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4834.332808] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4838.915225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4838.957784] Lustre: 284978:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4838.981132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4838.982687] Lustre: Skipped 1 previous similar message [ 4839.020590] Lustre: lustre-MDT0001: new disk, initializing [ 4839.053572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4839.066458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4839.071886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4840.344659] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4843.808039] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4844.982678] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4847.195328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4847.303261] Lustre: lustre-OST0000: new disk, initializing [ 4847.305559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4847.336224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4849.247355] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4852.511111] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4854.263039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4854.266057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4854.293988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4864.480524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4864.482627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4864.483596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4864.485742] Lustre: Skipped 1 previous similar message [ 4867.858277] Lustre: server umount lustre-OST0000 complete [ 4869.600426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4869.600868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4869.605076] Lustre: Skipped 1 previous similar message [ 4869.606764] Lustre: Skipped 1 previous similar message [ 4874.720059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4874.722167] Lustre: Skipped 2 previous similar messages [ 4875.173303] Lustre: server umount lustre-MDT0000 complete [ 4876.343223] LustreError: 284970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775171502 with bad export cookie 17849407922278656336 [ 4876.346194] LustreError: 284970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4876.346983] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4876.466815] Lustre: server umount lustre-MDT0001 complete [ 4879.745055] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4880.657740] Key type lgssc unregistered [ 4880.772398] LNet: 288350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4880.774889] LNetError: 288350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4880.783278] LNet: Removed LNI 192.168.202.159@tcp [ 4881.046875] Key type .llcrypt unregistered [ 4881.047986] Key type ._llcrypt unregistered [ 4890.038084] Key type ._llcrypt registered [ 4890.039114] Key type .llcrypt registered [ 4890.079682] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4890.392420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4890.420051] alg: No test for adler32 (adler32-zlib) [ 4891.265720] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 4891.350168] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4892.927118] Key type lgssc registered [ 4893.259452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4895.729241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4895.736395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4896.890669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4898.200293] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4900.976576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4901.091478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4902.397727] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4904.106020] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4905.239614] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4907.368794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4907.478598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4909.186617] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4910.819177] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4914.342400] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4914.836120] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4916.703817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4916.705807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.709198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4917.446263] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 4920.801165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4920.803917] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.810795] Lustre: Skipped 1 previous similar message [ 4922.326880] Lustre: server umount lustre-MDT0000 complete [ 4922.558885] LustreError: 289721:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.564064] LustreError: 289721:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4923.594194] LustreError: 289706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775171549 with bad export cookie 3775129775563091666 [ 4923.597250] LustreError: 289706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4923.597569] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4925.919848] LustreError: 289722:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4925.920123] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4925.927620] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4925.929491] Lustre: Skipped 2 previous similar messages [ 4927.678639] LustreError: 291150:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4929.763693] Lustre: server umount lustre-MDT0001 complete [ 4945.375177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4945.449408] Lustre: server umount lustre-OST0000 complete [ 4948.826955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.271063] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4951.597624] Lustre: 293453:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.159@tcp [ 4952.286703] Lustre: server umount MGS complete [ 4955.715174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4955.864285] LustreError: 294022:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4955.869505] LustreError: 294022:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4955.885452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4957.212261] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4960.343960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4960.490674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4962.037500] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4964.848141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4966.835394] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4968.957689] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4969.614581] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 4971.107268] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4971.675937] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4976.611733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4976.611913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4976.613470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4976.613476] Lustre: Skipped 2 previous similar messages [ 4976.628212] Lustre: Skipped 1 previous similar message [ 4979.862714] Lustre: server umount lustre-OST0000 complete [ 4981.727813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.336300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4986.340304] LustreError: Skipped 1 previous similar message [ 4986.343463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4986.345982] Lustre: Skipped 2 previous similar messages [ 4987.418783] Lustre: server umount lustre-MDT0000 complete [ 4988.905879] LustreError: 294003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775171614 with bad export cookie 3775129775563092492 [ 4988.908908] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4988.909227] LustreError: 294003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4989.022975] Lustre: server umount lustre-MDT0001 complete [ 4992.702065] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4993.701602] Key type lgssc unregistered [ 4993.821341] LNet: 297197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4993.823685] LNetError: 297197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4993.832273] LNet: Removed LNI 192.168.202.159@tcp [ 4994.134152] Key type .llcrypt unregistered [ 4994.135288] Key type ._llcrypt unregistered [ 5000.046811] Key type ._llcrypt registered [ 5000.047976] Key type .llcrypt registered [ 5000.092209] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 5006.932117] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5007.257157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5007.287445] alg: No test for adler32 (adler32-zlib) [ 5008.132178] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 5008.208463] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5009.783095] Key type lgssc registered [ 5010.221245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5023.474145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5023.479277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5024.571907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5024.583731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5024.622167] Lustre: lustre-MDT0000: new disk, initializing [ 5024.645683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5024.652254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5026.060849] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5030.633739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5030.684186] Lustre: 300292:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5030.710349] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5030.712829] Lustre: Skipped 1 previous similar message [ 5030.753900] Lustre: lustre-MDT0001: new disk, initializing [ 5030.785484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5030.797378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5030.801861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5032.290365] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5035.356553] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5036.520729] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5038.773328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5038.880203] Lustre: lustre-OST0000: new disk, initializing [ 5038.882487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5038.911413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5040.829718] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5041.288320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5041.294744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5041.343373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5044.473002] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5046.011200] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5046.082257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5047.729230] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5047.804732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5051.874071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5051.876504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5051.879072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5051.879235] LustreError: Skipped 1 previous similar message [ 5051.887102] Lustre: Skipped 1 previous similar message [ 5054.735516] Lustre: server umount lustre-OST0000 complete [ 5056.480674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5056.485747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5056.495785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5056.500171] Lustre: Skipped 1 previous similar message [ 5062.113494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5062.118252] Lustre: Skipped 2 previous similar messages [ 5067.232065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5067.233631] Lustre: Skipped 1 previous similar message [ 5070.303104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5070.361751] Lustre: server umount lustre-MDT0000 complete [ 5071.501316] LustreError: 300285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775171697 with bad export cookie 39863820083470330 [ 5071.503436] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5071.504356] LustreError: 300285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5071.612950] Lustre: server umount lustre-MDT0001 complete [ 5074.200550] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:15:00 (1775171700) [ 5078.999293] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 5081.787018] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5092.440043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5092.524870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5092.535939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5092.570748] Lustre: lustre-MDT0000: new disk, initializing [ 5092.592461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5092.598516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5093.806442] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5097.734469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5097.773059] Lustre: 306600:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5097.796556] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5097.798209] Lustre: Skipped 1 previous similar message [ 5097.834323] Lustre: lustre-MDT0001: new disk, initializing [ 5097.862418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5097.872239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5097.874949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5099.063511] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5101.965206] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5103.077076] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5105.114614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5106.951752] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5109.914489] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5111.285570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5111.289283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5111.319798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5126.624597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5126.626119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5126.626579] LustreError: Skipped 1 previous similar message [ 5126.629514] Lustre: Skipped 1 previous similar message [ 5126.630065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5126.635931] Lustre: Skipped 1 previous similar message [ 5130.092190] Lustre: server umount lustre-OST0000 complete [ 5131.744264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5131.750641] Lustre: Skipped 2 previous similar messages [ 5136.864172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5136.866298] Lustre: Skipped 3 previous similar messages [ 5137.591647] Lustre: server umount lustre-MDT0000 complete [ 5139.280777] LustreError: 307375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775171764 with bad export cookie 39863820083471198 [ 5139.285868] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5139.287993] LustreError: 307375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5139.418314] Lustre: server umount lustre-MDT0001 complete [ 5142.808921] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5143.728398] Key type lgssc unregistered [ 5143.851575] LNet: 310360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5143.856243] LNetError: 310360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5143.871425] LNet: Removed LNI 192.168.202.159@tcp [ 5144.250972] Key type .llcrypt unregistered [ 5144.252234] Key type ._llcrypt unregistered [ 5176.392236] Key type ._llcrypt registered [ 5176.393482] Key type .llcrypt registered [ 5176.433147] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5176.528371] Key type .llcrypt unregistered [ 5176.529662] Key type ._llcrypt unregistered [ 5206.149197] Key type ._llcrypt registered [ 5206.150215] Key type .llcrypt registered [ 5206.187100] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5206.297282] Key type .llcrypt unregistered [ 5206.298692] Key type ._llcrypt unregistered [ 5217.592494] Key type ._llcrypt registered [ 5217.594200] Key type .llcrypt registered [ 5217.642077] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5217.785281] Key type .llcrypt unregistered [ 5217.787070] Key type ._llcrypt unregistered [ 5230.588265] Key type ._llcrypt registered [ 5230.589507] Key type .llcrypt registered [ 5230.633311] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5230.733172] Key type .llcrypt unregistered [ 5230.734096] Key type ._llcrypt unregistered [ 5263.092203] Key type ._llcrypt registered [ 5263.094131] Key type .llcrypt registered [ 5263.139358] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5263.262458] Key type .llcrypt unregistered [ 5263.264145] Key type ._llcrypt unregistered [ 5276.856396] Key type ._llcrypt registered [ 5276.858457] Key type .llcrypt registered [ 5276.912968] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5277.081451] Key type .llcrypt unregistered [ 5277.084624] Key type ._llcrypt unregistered [ 5290.271632] Key type ._llcrypt registered [ 5290.272599] Key type .llcrypt registered [ 5290.304911] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5290.388325] Key type .llcrypt unregistered [ 5290.389772] Key type ._llcrypt unregistered [ 5302.871926] Key type ._llcrypt registered [ 5302.873107] Key type .llcrypt registered [ 5302.909623] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5303.011650] Key type .llcrypt unregistered [ 5303.013111] Key type ._llcrypt unregistered [ 5317.140107] Key type ._llcrypt registered [ 5317.141692] Key type .llcrypt registered [ 5317.215428] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5317.384871] Key type .llcrypt unregistered [ 5317.386608] Key type ._llcrypt unregistered [ 5329.782928] Key type ._llcrypt registered [ 5329.784299] Key type .llcrypt registered [ 5329.820198] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5329.909219] Key type .llcrypt unregistered [ 5329.910432] Key type ._llcrypt unregistered [ 5350.380962] Key type ._llcrypt registered [ 5350.381994] Key type .llcrypt registered [ 5350.426267] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5350.527451] Key type .llcrypt unregistered [ 5350.528572] Key type ._llcrypt unregistered [ 5361.427615] Key type ._llcrypt registered [ 5361.429154] Key type .llcrypt registered [ 5361.462738] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5361.557348] Key type .llcrypt unregistered [ 5361.558394] Key type ._llcrypt unregistered [ 5373.277301] Key type ._llcrypt registered [ 5373.278413] Key type .llcrypt registered [ 5373.313539] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5373.408220] Key type .llcrypt unregistered [ 5373.409524] Key type ._llcrypt unregistered [ 5393.836966] Key type ._llcrypt registered [ 5393.838382] Key type .llcrypt registered [ 5393.881570] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 5400.635577] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5400.924904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5400.957065] alg: No test for adler32 (adler32-zlib) [ 5401.798351] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 5401.884593] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5403.471099] Key type lgssc registered [ 5403.859515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5415.318551] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5270 sec ========== 19:20:41 (1775172041) [ 5416.070477] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:20:42 (1775172042) === [ 5417.459515] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:20:43 (1775172043) === [ 5432.506517] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5433.645341] Key type lgssc unregistered [ 5433.767288] LNet: 331946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5433.769120] LNetError: 331946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5433.784269] LNet: Removed LNI 192.168.202.159@tcp [ 5434.019364] Key type .llcrypt unregistered [ 5434.020285] Key type ._llcrypt unregistered