[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478524866 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002346] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007375] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010140] LSM: Security Framework initializing [ 0.012008] Yama: becoming mindful. [ 0.012990] SELinux: Initializing. [ 0.014073] *** VALIDATE selinux *** [ 0.022099] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027235] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.032367] *** VALIDATE proc *** [ 0.033242] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.035265] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.041006] Speculative Store Bypass: Vulnerable [ 0.044008] debug: unmapping init [mem 0xffffffff8ac59000-0xffffffff8ac60fff] [ 0.046191] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047650] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050013] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053012] ... fixed-purpose events: 3 [ 0.054011] ... event mask: 000000070000000f [ 0.055308] rcu: Hierarchical SRCU implementation. [ 0.057502] smp: Bringing up secondary CPUs ... [ 0.058527] x86: Booting SMP configuration: [ 0.059026] .... node #0, CPUs: #1 #2 #3 [ 0.062389] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135117] node 0 deferred pages initialised in 67ms [ 0.139214] devtmpfs: initialized [ 0.140239] x86/mm: Memory block size: 128MB [ 0.142783] gcov: version magic: 0x41383552 [ 0.144581] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151338] pinctrl core: initialized pinctrl subsystem [ 0.154184] [ 0.154781] ************************************************************* [ 0.157013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159011] ** ** [ 0.161013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163013] ** ** [ 0.165018] ** This means that this kernel is built to expose internal ** [ 0.167012] ** IOMMU data structures, which may compromise security on ** [ 0.169012] ** your system. ** [ 0.171016] ** ** [ 0.174015] ** If you see this message and you are not debugging the ** [ 0.177026] ** kernel, report this immediately to your vendor! ** [ 0.179016] ** ** [ 0.181013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184013] ************************************************************* [ 0.187830] NET: Registered protocol family 16 [ 0.189413] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201073] cpuidle: using governor menu [ 0.202584] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.206444] PCI: Using configuration type 1 for base access [ 0.209164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220075] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223177] cryptd: max_cpu_qlen set to 1000 [ 0.227303] ACPI: Added _OSI(Module Device) [ 0.229015] ACPI: Added _OSI(Processor Device) [ 0.230014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237271] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.244139] ACPI: Interpreter enabled [ 0.245056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.248035] ACPI: Using IOAPIC for interrupt routing [ 0.250182] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.254625] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.268049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279475] acpiphp: Slot [2] registered [ 0.281112] acpiphp: Slot [5] registered [ 0.282119] acpiphp: Slot [6] registered [ 0.283111] acpiphp: Slot [7] registered [ 0.285135] acpiphp: Slot [8] registered [ 0.287127] acpiphp: Slot [9] registered [ 0.288142] acpiphp: Slot [10] registered [ 0.290167] acpiphp: Slot [3] registered [ 0.292198] acpiphp: Slot [4] registered [ 0.293144] acpiphp: Slot [11] registered [ 0.295170] acpiphp: Slot [12] registered [ 0.297158] acpiphp: Slot [13] registered [ 0.299149] acpiphp: Slot [14] registered [ 0.301109] acpiphp: Slot [15] registered [ 0.303131] acpiphp: Slot [16] registered [ 0.304117] acpiphp: Slot [17] registered [ 0.306101] acpiphp: Slot [18] registered [ 0.307105] acpiphp: Slot [19] registered [ 0.308101] acpiphp: Slot [20] registered [ 0.309108] acpiphp: Slot [21] registered [ 0.311133] acpiphp: Slot [22] registered [ 0.313170] acpiphp: Slot [23] registered [ 0.315224] acpiphp: Slot [24] registered [ 0.316187] acpiphp: Slot [25] registered [ 0.318155] acpiphp: Slot [26] registered [ 0.320171] acpiphp: Slot [27] registered [ 0.322159] acpiphp: Slot [28] registered [ 0.323161] acpiphp: Slot [29] registered [ 0.325199] acpiphp: Slot [30] registered [ 0.327154] acpiphp: Slot [31] registered [ 0.329196] PCI host bridge to bus 0000:00 [ 0.330025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.333026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.335023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.338028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.341031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.344029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.346188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353605] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.364806] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.370056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.375018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.378021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.381563] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.384077] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.392128] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.398018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.413015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.417015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.423946] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.431018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.457021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.466581] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.475013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.488017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.509018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.524382] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.535016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.543014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.561015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.571360] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.578000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.586020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.610019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.624780] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.631018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.638018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.656016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.668881] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.674015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.680017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.699013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.710090] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.716573] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.722373] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.725424] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.728261] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.734096] iommu: Default domain type: Passthrough [ 0.735546] SCSI subsystem initialized [ 0.737188] ACPI: bus type USB registered [ 0.739175] usbcore: registered new interface driver usbfs [ 0.740067] usbcore: registered new interface driver hub [ 0.742085] usbcore: registered new device driver usb [ 0.743157] pps_core: LinuxPPS API ver. 1 registered [ 0.745013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.749067] PTP clock support registered [ 0.751134] EDAC MC: Ver: 3.0.0 [ 0.753130] PCI: Using ACPI for IRQ routing [ 0.755793] NetLabel: Initializing [ 0.757013] NetLabel: domain hash size = 128 [ 0.758009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.760109] NetLabel: unlabeled traffic allowed by default [ 0.763043] vgaarb: loaded [ 0.764286] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.765008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.770326] clocksource: Switched to clocksource kvm-clock [ 0.877603] VFS: Disk quotas dquot_6.6.0 [ 0.879031] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.884302] *** VALIDATE ramfs *** [ 0.885662] *** VALIDATE hugetlbfs *** [ 0.886879] pnp: PnP ACPI init [ 0.889198] pnp: PnP ACPI: found 6 devices [ 0.925531] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.929354] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.931635] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.933905] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.936596] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.941684] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.944880] NET: Registered protocol family 2 [ 0.947384] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.952637] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.956623] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.962089] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.965784] TCP: Hash tables configured (established 65536 bind 65536) [ 0.969238] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.972848] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.975890] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.979507] NET: Registered protocol family 1 [ 0.982902] RPC: Registered named UNIX socket transport module. [ 0.984962] RPC: Registered udp transport module. [ 0.987288] RPC: Registered tcp transport module. [ 0.989030] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.991509] NET: Registered protocol family 44 [ 0.993231] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.995497] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.997521] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.999946] PCI: CLS 0 bytes, default 64 [ 1.002344] Unpacking initramfs... [ 2.414702] debug: unmapping init [mem 0xffff9803bcc54000-0xffff9803bffbffff] [ 2.418817] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.421115] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.423917] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.934889] Initialise system trusted keyrings [ 2.936928] Key type blacklist registered [ 2.938397] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.949278] zbud: loaded [ 2.952688] *** VALIDATE nfs *** [ 2.954110] *** VALIDATE nfs4 *** [ 2.955382] pstore: using deflate compression [ 2.959479] Platform Keyring initialized [ 3.062201] NET: Registered protocol family 38 [ 3.063886] Key type asymmetric registered [ 3.065642] Asymmetric key parser 'x509' registered [ 3.067685] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.070762] io scheduler mq-deadline registered [ 3.072649] io scheduler kyber registered [ 3.074324] io scheduler bfq registered [ 3.077132] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.080172] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.083145] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.085948] ACPI: Power Button [PWRF] [ 3.091103] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.097742] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.109453] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.116260] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.129055] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.157809] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.186794] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.191611] Non-volatile memory driver v1.3 [ 3.194132] Linux agpgart interface v0.103 [ 3.225331] virtio_blk virtio1: [vda] 146728 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.228341] vda: detected capacity change from 0 to 75124736 [ 3.243638] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.247018] vdb: detected capacity change from 0 to 1073741824 [ 3.261441] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.264642] vdc: detected capacity change from 0 to 2621440000 [ 3.283708] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.287532] vdd: detected capacity change from 0 to 2621440000 [ 3.305328] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.308255] vde: detected capacity change from 0 to 4294967296 [ 3.322969] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.326387] vdf: detected capacity change from 0 to 4294967296 [ 3.332209] libphy: Fixed MDIO Bus: probed [ 3.337729] usbcore: registered new interface driver usbserial_generic [ 3.340576] usbserial: USB Serial support registered for generic [ 3.343954] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.348792] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.350846] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.353687] mousedev: PS/2 mouse device common for all mice [ 3.357585] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.362031] rtc_cmos 00:05: RTC can wake from S4 [ 3.365059] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.367903] rtc_cmos 00:05: registered as rtc0 [ 3.372078] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.372192] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.379455] intel_pstate: CPU model not supported [ 3.382812] hid: raw HID events driver (C) Jiri Kosina [ 3.384980] usbcore: registered new interface driver usbhid [ 3.386996] usbhid: USB HID core driver [ 3.388551] drop_monitor: Initializing network drop monitor service [ 3.390378] Initializing XFRM netlink socket [ 3.392528] NET: Registered protocol family 10 [ 3.395578] Segment Routing with IPv6 [ 3.397096] NET: Registered protocol family 17 [ 3.399467] mpls_gso: MPLS GSO support [ 3.405763] RAS: Correctable Errors collector initialized. [ 3.408792] AVX version of gcm_enc/dec engaged. [ 3.410815] AES CTR mode by8 optimization enabled [ 3.488479] sched_clock: Marking stable (3488457377, 0)->(4350271915, -861814538) [ 3.492277] registered taskstats version 1 [ 3.494480] Loading compiled-in X.509 certificates [ 3.496965] zswap: loaded using pool lzo/zbud [ 3.525554] Key type big_key registered [ 3.540233] Key type encrypted registered [ 3.542014] ima: No TPM chip found, activating TPM-bypass! [ 3.543992] ima: Allocated hash algorithm: sha1 [ 3.545948] ima: No architecture policies found [ 3.547866] evm: Initialising EVM extended attributes: [ 3.549903] evm: security.selinux [ 3.551884] evm: security.ima [ 3.553285] evm: security.capability [ 3.554751] evm: HMAC attrs: 0x1 [ 3.557216] rtc_cmos 00:05: setting system clock to 2026-09-08 03:05:24 UTC (1788836724) [ 3.563520] debug: unmapping init [mem 0xffffffff8bc03000-0xffffffff8bdfffff] [ 3.566908] debug: unmapping init [mem 0xffffffff8a982000-0xffffffff8ac58fff] [ 3.575247] Write protecting the kernel read-only data: 28672k [ 3.578842] debug: unmapping init [mem 0xffffffff89003000-0xffffffff891fffff] [ 3.581983] debug: unmapping init [mem 0xffffffff89914000-0xffffffff899fffff] [ 3.617483] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.626749] systemd[1]: Detected virtualization kvm. [ 3.628970] systemd[1]: Detected architecture x86-64. [ 3.630474] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.655516] systemd[1]: No hostname configured. [ 3.656997] systemd[1]: Set hostname to . [ 3.658826] random: systemd: uninitialized urandom read (16 bytes read) [ 3.660492] systemd[1]: Initializing machine ID from random generator. [ 3.711470] random: ln: uninitialized urandom read (6 bytes read) [ 3.825988] random: systemd: uninitialized urandom read (16 bytes read) [ 3.828785] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.832958] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.837316] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.386252] device-mapper: uevent: version 1.0.3 [ 4.388044] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.170359] virtio_net virtio0 ens2: renamed from eth0 [ 5.190395] random: fast init done [ 5.250152] scsi host0: ata_piix [ 5.259778] scsi host1: ata_piix [ 5.261604] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.264493] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.600861] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.062504] random: crng init done [ 10.064153] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.551831] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.866189] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.179185] SELinux: Disabled at runtime. [ 12.242184] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.253433] systemd[1]: Detected virtualization kvm. [ 12.255337] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.824979] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.827910] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.832444] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.836086] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.840201] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.846229] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.854415] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 12.954324] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.419398] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.768450] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.822831] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.959486] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.970772] EDAC sbridge: Ver: 1.1.2 [ 16.148396] Key type dns_resolver registered [ 16.471192] NFS: Registering the id_resolver key type [ 16.474823] Key type id_resolver registered [ 16.476465] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg416-server login: [ 39.323185] hrtimer: interrupt took 4198185 ns [ 52.819428] libcfs: loading out-of-tree module taints kernel. [ 52.855797] Key type ._llcrypt registered [ 52.859940] Key type .llcrypt registered [ 53.033751] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 75.915319] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 78.206925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 78.251739] alg: No test for adler32 (adler32-zlib) [ 80.036891] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 81.060470] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 82.896240] Key type lgssc registered [ 85.338896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.732949] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 158.022548] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 175.758322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 175.795271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.159274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 177.219255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 177.366424] Lustre: lustre-MDT0000: new disk, initializing [ 177.481780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.514790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 182.960234] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.993148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.169337] Lustre: 6525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 199.235208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 199.250735] Lustre: Skipped 1 previous similar message [ 199.392271] Lustre: lustre-MDT0001: new disk, initializing [ 199.495499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.532029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 199.548245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 205.307785] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.413573] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 227.148355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 227.459252] Lustre: lustre-OST0000: new disk, initializing [ 227.463528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 227.471029] Lustre: 8464:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 227.545414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.004345] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.054234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 234.073600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 234.176749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 252.319329] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 252.529900] Lustre: lustre-OST0001: new disk, initializing [ 252.533175] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 252.546640] Lustre: 9537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 252.629658] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 259.170410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 259.185451] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 259.255966] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 260.102571] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.554826] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 281.191418] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 288.576264] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing check_logdir /tmp/testlogs/ [ 294.294451] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing yml_node [ 301.197868] Lustre: DEBUG MARKER: Client: 2.17.57.113 [ 304.392353] Lustre: DEBUG MARKER: MDS: 2.17.57.113 [ 308.441724] Lustre: DEBUG MARKER: OSS: 2.17.57.113 [ 311.868627] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 23:10:30 EDT 2026 [ 331.228802] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 333.275951] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 349.664353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.666458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 349.684272] Lustre: Skipped 1 previous similar message [ 354.787563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.795628] Lustre: Skipped 6 previous similar messages [ 355.940962] Lustre: server umount lustre-MDT0000 complete [ 359.835729] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837080 with bad export cookie 13339242288740864837 [ 359.840406] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 359.857034] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 359.913513] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.918068] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 359.930161] Lustre: Skipped 3 previous similar messages [ 359.944333] Lustre: Skipped 1 previous similar message [ 365.024379] LustreError: 6533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.025747] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 365.070317] LustreError: 6533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 365.101735] Lustre: Skipped 1 previous similar message [ 369.120323] LustreError: 6532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.121312] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 369.157756] LustreError: 6532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 369.176843] Lustre: Skipped 1 previous similar message [ 374.239184] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 374.255552] LustreError: 9396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.293354] LustreError: 9396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 374.551644] Lustre: server umount lustre-MDT0001 complete [ 378.888471] Lustre: server umount lustre-OST0000 complete [ 383.769464] Lustre: server umount lustre-OST0001 complete [ 392.761848] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 404.601641] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 447.297478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.689490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 447.759553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 447.961154] Lustre: lustre-MDT0000: new disk, initializing [ 448.111568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.169382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 453.096649] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 465.795443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.903388] Lustre: 15093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 466.022629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 466.032244] Lustre: Skipped 1 previous similar message [ 466.154695] Lustre: lustre-MDT0001: new disk, initializing [ 466.278119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 466.339625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 466.350320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 471.166706] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 480.682940] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 486.172145] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 493.894487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 494.284503] Lustre: lustre-OST0000: new disk, initializing [ 494.289252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 494.304529] Lustre: 17063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 494.471462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.756659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 495.776610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 495.894282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 500.890662] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 511.570288] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 517.257963] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 517.561664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 522.148881] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 522.436323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 526.816567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 526.829269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.847871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 526.856701] Lustre: Skipped 3 previous similar messages [ 530.766110] Lustre: server umount lustre-OST0000 complete [ 537.060079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.076459] Lustre: Skipped 2 previous similar messages [ 541.275661] Lustre: server umount lustre-MDT0000 complete [ 542.178974] LustreError: 15936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.206148] LustreError: 15936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 545.811264] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837266 with bad export cookie 13339242288740866370 [ 545.824467] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 545.827060] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 546.267815] Lustre: server umount lustre-MDT0001 complete [ 552.663044] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 23:14:32 (1788837272) [ 571.132516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.584733] LustreError: 19936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.630488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 576.334094] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 581.104370] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.118662] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 584.925919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 589.563596] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 596.338157] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 602.213399] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 604.199980] Lustre: Failing over lustre-MDT0000 [ 604.461113] Lustre: server umount lustre-MDT0000 complete [ 605.665119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 605.670733] LustreError: Skipped 1 previous similar message [ 605.673751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.697402] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.074484] Lustre: 3638:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788837327/real 1788837327] req@ffff9803035ed880 x1875731337614592/t0(0) o400->MGC192.168.204.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788837343 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 623.099705] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 625.862410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.329577] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb91e822ff1449ecf [ 633.353061] Lustre: MGC192.168.204.116@tcp: Connection restored to 0@lo (at 0@lo) [ 633.896938] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 638.955539] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 638.980281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 639.010568] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 639.821122] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 651.811480] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 659.426094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 659.438260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 659.450915] Lustre: Skipped 1 previous similar message [ 659.455576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 659.460656] Lustre: Skipped 2 previous similar messages [ 662.742954] Lustre: server umount lustre-MDT0000 complete [ 665.575271] LustreError: 19932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 665.605841] LustreError: 19932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 668.289757] LustreError: 19917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837389 with bad export cookie 13339242288740867791 [ 668.292636] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 668.296962] LustreError: 19917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 668.778793] Lustre: server umount lustre-MDT0001 complete [ 680.992454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.869578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 681.883804] Lustre: Skipped 2 previous similar messages [ 687.855453] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 699.157110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 706.472615] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 714.339987] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 720.978397] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 723.158287] Lustre: Failing over lustre-MDT0000 [ 723.504988] Lustre: server umount lustre-MDT0000 complete [ 724.204256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 724.224541] Lustre: Skipped 1 previous similar message [ 729.572718] LustreError: 24008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 729.596764] LustreError: 24008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 740.384141] Lustre: 3636:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788837445/real 1788837445] req@ffff980441ac3480 x1875731337646464/t0(0) o400->MGC192.168.204.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788837461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 740.430927] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 743.640957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 750.688050] LustreError: 3635:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff98042e0d9880 x1875731337649408/t0(0) o250->MGC192.168.204.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 751.052490] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 756.215713] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 756.244308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 756.254879] Lustre: Skipped 1 previous similar message [ 756.294287] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 757.330937] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 768.377578] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 775.138149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.142562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 775.165113] Lustre: Skipped 2 previous similar messages [ 775.185160] Lustre: Skipped 3 previous similar messages [ 778.695645] Lustre: server umount lustre-MDT0000 complete [ 782.528625] LustreError: 23257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837503 with bad export cookie 13339242288740868729 [ 782.543939] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 782.909181] Lustre: server umount lustre-MDT0001 complete [ 791.819894] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 23:18:32 (1788837512) [ 808.929139] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 819.436244] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 862.373828] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 876.321535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 876.757874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 876.836624] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 877.006773] Lustre: lustre-MDT0000: new disk, initializing [ 877.172311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.188955] Lustre: Skipped 2 previous similar messages [ 877.233065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 882.149577] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 893.449017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.610519] Lustre: 29658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 893.912495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 899.788978] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 911.229031] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 918.142157] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 925.890686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 926.302094] Lustre: 31628:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 927.549354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 927.584926] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 927.653624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 927.668215] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 933.061876] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 944.714081] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 951.336513] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 973.796428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 973.805072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.808050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 973.822768] Lustre: Skipped 1 previous similar message [ 973.837411] Lustre: Skipped 1 previous similar message [ 977.187452] Lustre: server umount lustre-OST0000 complete [ 988.011802] Lustre: server umount lustre-MDT0000 complete [ 989.155787] LustreError: 29669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.192827] LustreError: 29669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 993.132206] LustreError: 29649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837714 with bad export cookie 13339242288740869135 [ 993.135920] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 993.156921] LustreError: 29649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 993.893977] Lustre: server umount lustre-MDT0001 complete [ 1007.246165] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1010.925751] Key type lgssc unregistered [ 1011.268800] LNet: 33607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.282574] LNetError: 33607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.304849] LNet: Removed LNI 192.168.204.116@tcp [ 1012.767145] Key type .llcrypt unregistered [ 1012.769143] Key type ._llcrypt unregistered [ 1039.941686] Key type ._llcrypt registered [ 1039.949252] Key type .llcrypt registered [ 1040.098416] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1041.181222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1041.275260] alg: No test for adler32 (adler32-zlib) [ 1042.603584] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 1042.876874] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1044.543465] Key type lgssc registered [ 1046.080229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1054.587709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1054.623992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1056.132308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1061.621341] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1066.469991] LustreError: 35019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1066.490924] LustreError: 35019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1070.574131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1071.110493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1076.039619] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1083.899341] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1090.651511] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1099.112722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1099.684562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1101.796318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1106.796418] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1115.470789] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1116.084360] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1130.712549] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 23:24:11 (1788837851) [ 1132.759550] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1134.675706] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1141.547458] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 23:24:21 (1788837861) [ 1150.548309] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 23:24:30 (1788837870) [ 1151.763808] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1161.172745] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1161.178269] Lustre: Skipped 2 previous similar messages [ 1162.914727] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 23:24:43 (1788837883) [ 1166.679153] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1166.684460] Lustre: Skipped 3 previous similar messages [ 1187.175699] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1187.183425] Lustre: Skipped 6 previous similar messages [ 1190.754781] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 23:25:10 (1788837910) [ 1193.603598] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1197.337892] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1206.274488] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 23:25:26 (1788837926) [ 1207.859317] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1207.867203] Lustre: Skipped 3 previous similar messages [ 1221.246799] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 23:25:41 (1788837941) [ 1239.016248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1239.021184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.023890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1239.051311] Lustre: Skipped 1 previous similar message [ 1242.920740] Lustre: server umount lustre-OST0000 complete [ 1249.254630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.313218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1249.322437] Lustre: Skipped 2 previous similar messages [ 1254.041894] Lustre: server umount lustre-MDT0000 complete [ 1254.369315] LustreError: 35747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.409737] LustreError: 35747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1258.817694] LustreError: 35001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837979 with bad export cookie 12867763074482216548 [ 1258.820372] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1258.829590] LustreError: 35001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1259.269788] Lustre: server umount lustre-MDT0001 complete [ 1273.713190] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1277.644795] Key type lgssc unregistered [ 1278.053964] LNet: 42883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.070750] LNetError: 42883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1278.095312] LNet: Removed LNI 192.168.204.116@tcp [ 1279.160614] Key type .llcrypt unregistered [ 1279.169778] Key type ._llcrypt unregistered [ 1298.174387] Key type ._llcrypt registered [ 1298.176904] Key type .llcrypt registered [ 1298.343683] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 1315.290154] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1316.470060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1316.593844] alg: No test for adler32 (adler32-zlib) [ 1317.822970] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 1318.154756] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1319.903936] Key type lgssc registered [ 1320.982859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1363.198885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1363.244425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.472448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1364.508113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1364.629285] Lustre: lustre-MDT0000: new disk, initializing [ 1364.797275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1364.817953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1369.869073] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1380.018166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.147307] Lustre: 46035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1380.240740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1380.246487] Lustre: Skipped 1 previous similar message [ 1380.354130] Lustre: lustre-MDT0001: new disk, initializing [ 1380.450689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1380.482200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1380.487509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1385.531977] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1395.262658] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1402.267742] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1410.904458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1411.253452] Lustre: lustre-OST0000: new disk, initializing [ 1411.262317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1411.276547] Lustre: 48004:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1411.441160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.838451] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1419.914374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1419.935747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1420.108412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1429.021773] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1434.656844] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1434.903668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1439.640146] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1439.879637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1445.860672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1445.866333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.869578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.891946] Lustre: Skipped 1 previous similar message [ 1448.316399] Lustre: server umount lustre-OST0000 complete [ 1456.098106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.111931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.122798] Lustre: Skipped 2 previous similar messages [ 1458.598887] Lustre: server umount lustre-MDT0000 complete [ 1463.098648] LustreError: 46027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838184 with bad export cookie 7783126988096268315 [ 1463.102898] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1463.111263] LustreError: 46027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1463.404653] Lustre: server umount lustre-MDT0001 complete [ 1472.893655] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 23:29:52 (1788838192) [ 1479.630357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.128537] LustreError: 50465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1480.158717] LustreError: 50465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1480.232737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.288408] LustreError: 50466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1485.541596] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1490.402240] LustreError: 50465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1494.392279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1494.862246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1499.618754] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1509.480965] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1515.399565] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1521.839062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1522.350273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1529.999446] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1537.755387] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1545.444507] LustreError: 52995:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1546.409041] Lustre: Modifying parameter general.timeout=1 in log params [ 1547.981730] Lustre: Modifying parameter general.timeout=3 in log params [ 1547.985432] Lustre: Skipped 1 previous similar message [ 1550.832973] Lustre: Modifying parameter general.timeout=6 in log params [ 1550.839761] Lustre: Skipped 2 previous similar messages [ 1555.754811] Lustre: Modifying parameter general.timeout=11 in log params [ 1555.761757] Lustre: Skipped 4 previous similar messages [ 1564.230592] Lustre: Modifying parameter general.timeout=20 in log params [ 1564.234327] Lustre: Skipped 8 previous similar messages [ 1580.462543] Lustre: Modifying parameter general.timeout=40 in log params [ 1580.466908] Lustre: Skipped 19 previous similar messages [ 1612.695596] Lustre: Modifying parameter general.timeout=77 in log params [ 1612.699414] Lustre: Skipped 36 previous similar messages [ 1678.272654] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 23:33:18 (1788838398) [ 1679.404447] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1679.410645] Lustre: Skipped 53 previous similar messages [ 1681.399687] Lustre: 59945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1683.875408] Lustre: 60045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1686.623610] Lustre: 60156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1691.808617] Lustre: 60353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1691.827157] Lustre: 60353:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1726.483910] Lustre: 61822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1726.498337] Lustre: 61822:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1726.526493] LustreError: 61822:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1730.887760] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 23:34:11 (1788838451) [ 1745.376794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.386100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1745.398796] Lustre: Skipped 3 previous similar messages [ 1745.408455] Lustre: Skipped 1 previous similar message [ 1750.499453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1750.517783] Lustre: Skipped 3 previous similar messages [ 1751.116181] Lustre: server umount lustre-MDT0000 complete [ 1754.945361] LustreError: 50447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838475 with bad export cookie 7783126988096269197 [ 1754.953862] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1754.964646] LustreError: 50447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1755.396142] Lustre: server umount lustre-MDT0001 complete [ 1769.495112] Lustre: server umount lustre-OST0000 complete [ 1798.161114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.380849] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1798.652200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.530306] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1813.237822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.320846] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1813.346327] Lustre: 64353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1818.709688] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1826.688981] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1832.073746] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1838.661514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1838.860184] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1839.190086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1839.202246] Lustre: Skipped 1 previous similar message [ 1846.166831] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1853.479391] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1855.201224] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1855.209219] Lustre: Skipped 22 previous similar messages [ 1855.253634] Lustre: 66918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1863.137943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1863.150086] LustreError: Skipped 1 previous similar message [ 1863.156321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.168741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1869.021750] Lustre: server umount lustre-OST0000 complete [ 1873.894956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.911665] Lustre: Skipped 1 previous similar message [ 1873.921213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1873.926806] Lustre: Skipped 4 previous similar messages [ 1879.010962] LustreError: 64365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1879.026396] LustreError: 64365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1879.159791] Lustre: server umount lustre-MDT0000 complete [ 1882.796062] LustreError: 64345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838603 with bad export cookie 7783126988096272368 [ 1882.806678] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1882.817787] LustreError: 64345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1883.154458] Lustre: server umount lustre-MDT0001 complete [ 1894.975906] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1897.902829] Key type lgssc unregistered [ 1898.341057] LNet: 68127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.355070] LNetError: 68127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1899.447921] LNet: Removed LNI 192.168.204.116@tcp [ 1900.557196] Key type .llcrypt unregistered [ 1900.559751] Key type ._llcrypt unregistered [ 1925.665088] Key type ._llcrypt registered [ 1925.667699] Key type .llcrypt registered [ 1925.815564] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1927.603990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1927.637348] alg: No test for adler32 (adler32-zlib) [ 1928.827694] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 1929.154449] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1930.925334] Key type lgssc registered [ 1932.392768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1942.157960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1942.190368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.735173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.028667] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1953.262764] LustreError: 69539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.279274] LustreError: 69539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1958.380529] LustreError: 69540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.697505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.351877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1965.390650] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1972.796820] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1977.899564] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1984.452204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1984.974865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.109639] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1999.736481] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2008.398707] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2018.785870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.804266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2018.809101] Lustre: Skipped 2 previous similar messages [ 2018.848019] Lustre: Skipped 2 previous similar messages [ 2023.905035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2023.918482] Lustre: Skipped 2 previous similar messages [ 2024.590547] Lustre: server umount lustre-MDT0000 complete [ 2028.541835] LustreError: 69520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838749 with bad export cookie 9008708726363688061 [ 2028.547581] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.555870] LustreError: 69520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2028.964180] Lustre: server umount lustre-MDT0001 complete [ 2043.720881] Lustre: server umount lustre-OST0000 complete [ 2070.186187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.574681] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2070.631554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2071.023608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.186549] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2086.178713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.309405] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2086.340852] Lustre: 74416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2086.810518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2091.120712] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2098.438693] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2103.882924] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2109.772299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2109.985307] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2110.369249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.091569] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2124.454665] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2126.562676] Lustre: 76987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2128.581089] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2136.545451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2136.555833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.572173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2139.809449] Lustre: server umount lustre-OST0000 complete [ 2147.297112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.299617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2147.323945] Lustre: Skipped 2 previous similar messages [ 2147.339492] Lustre: Skipped 3 previous similar messages [ 2150.118309] Lustre: server umount lustre-MDT0000 complete [ 2152.417597] LustreError: 75477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2152.443412] LustreError: 75477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2154.283032] LustreError: 76990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838875 with bad export cookie 9008708726363688908 [ 2154.292505] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2154.296445] LustreError: 76990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2154.667378] Lustre: server umount lustre-MDT0001 complete [ 2165.171116] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2168.618711] Key type lgssc unregistered [ 2168.989129] LNet: 78198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.005586] LNetError: 78198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.051300] LNet: Removed LNI 192.168.204.116@tcp [ 2170.358783] Key type .llcrypt unregistered [ 2170.362863] Key type ._llcrypt unregistered [ 2198.704670] Key type ._llcrypt registered [ 2198.710169] Key type .llcrypt registered [ 2198.817462] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2200.101172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2200.144947] alg: No test for adler32 (adler32-zlib) [ 2201.337392] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 2201.587870] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2203.255243] Key type lgssc registered [ 2204.584702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2214.004262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2214.030494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2215.858234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.483095] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2225.121060] LustreError: 79622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.142673] LustreError: 79622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2230.249769] LustreError: 79621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.028511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.726643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2238.825030] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2247.470557] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2253.701273] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2261.452127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2262.144736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.186384] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2277.821400] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2294.170873] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2295.285309] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2296.368140] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2298.448698] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2298.453227] Lustre: Skipped 1 previous similar message [ 2302.506482] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2302.511159] Lustre: Skipped 3 previous similar messages [ 2311.070243] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2311.074377] Lustre: Skipped 7 previous similar messages [ 2327.252047] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2327.262682] Lustre: Skipped 15 previous similar messages [ 2360.144443] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2360.153428] Lustre: Skipped 30 previous similar messages [ 2408.337937] Lustre: 87760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2411.314256] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 23:45:31 (1788839131) [ 2413.503451] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2415.823921] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 23:45:35 (1788839135) [ 2423.276222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2423.294258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.313326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2424.293516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2424.294551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2424.303180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.327434] Lustre: Skipped 1 previous similar message [ 2429.410079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2429.424313] Lustre: Skipped 1 previous similar message [ 2434.015406] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2434.239858] Lustre: server umount lustre-OST0000 complete [ 2439.650566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.658715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2439.674033] Lustre: Skipped 1 previous similar message [ 2444.769366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2444.780136] Lustre: Skipped 2 previous similar messages [ 2445.124397] Lustre: server umount lustre-MDT0000 complete [ 2449.036059] LustreError: 81473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839169 with bad export cookie 6158155644015409437 [ 2449.055325] LustreError: 81473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2449.062966] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2449.475917] Lustre: server umount lustre-MDT0001 complete [ 2460.413374] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2463.234682] Key type lgssc unregistered [ 2463.632740] LNet: 89008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2463.644915] LNetError: 89008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2463.667287] LNet: Removed LNI 192.168.204.116@tcp [ 2464.513815] Key type .llcrypt unregistered [ 2464.517419] Key type ._llcrypt unregistered [ 2469.181110] Key type ._llcrypt registered [ 2469.182915] Key type .llcrypt registered [ 2469.390932] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2475.174343] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules [ 2477.457498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2477.496165] alg: No test for adler32 (adler32-zlib) [ 2478.819570] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 2479.264592] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2481.029870] Key type lgssc registered [ 2482.591160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2484.128081] LustreError: 90254:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2502.622850] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2506.888139] LustreError: 90254:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2517.751492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2517.791235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.883182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.045695] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2529.250439] LustreError: 91251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.273274] LustreError: 91251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2534.371657] LustreError: 91250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.204208] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 23:47:35 (1788839255) [ 2539.491718] LustreError: 91251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.067416] LustreError: 91282:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2542.087299] LustreError: 91282:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2542.093500] LustreError: 91282:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2542.403455] Lustre: server umount lustre-MDT0000 complete [ 2559.348294] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2563.112459] Key type lgssc unregistered [ 2563.514124] LNet: 92775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2563.562246] LNetError: 92775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2563.589179] LNet: Removed LNI 192.168.204.116@tcp [ 2564.795594] Key type .llcrypt unregistered [ 2564.800612] Key type ._llcrypt unregistered [ 2584.359446] Key type ._llcrypt registered [ 2584.365151] Key type .llcrypt registered [ 2584.496811] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2585.654854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2585.792498] alg: No test for adler32 (adler32-zlib) [ 2586.918858] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 2587.240051] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2588.919261] Key type lgssc registered [ 2589.984880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2598.445853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2598.498400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.265246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.081888] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2610.664759] LustreError: 93989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.691866] LustreError: 93989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2615.412925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.177523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2621.116491] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2629.748511] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2636.043491] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2643.853656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2644.499494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.704880] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2661.021730] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2718.250871] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 23:50:38 (1788839438) [ 2720.336512] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2722.410274] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 23:50:42 (1788839442) [ 2727.400514] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2727.406031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2727.421064] Lustre: Skipped 2 previous similar messages [ 2727.441588] Lustre: Skipped 2 previous similar messages [ 2732.517266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2732.528584] Lustre: Skipped 2 previous similar messages [ 2733.240825] Lustre: server umount lustre-MDT0000 complete [ 2736.723582] LustreError: 93969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839457 with bad export cookie 16876527971683712159 [ 2736.741482] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2736.748331] LustreError: 93969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2737.216352] Lustre: server umount lustre-MDT0001 complete [ 2752.114537] Lustre: server umount lustre-OST0000 complete [ 2765.158979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.894788] LustreError: 98288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.089834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2771.440761] LustreError: 98289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.484376] LustreError: 98289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2771.564610] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2775.528210] LustreError: 98288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.644760] LustreError: 98289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.120897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2783.035285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2787.714689] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2796.613785] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2803.396515] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2817.646724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2818.047722] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2818.057307] LustreError: 98278:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2818.090857] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2818.099775] LustreError: 100373:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2818.105852] LustreError: 100373:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2818.118072] LustreError: 100373:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2818.245712] Lustre: server umount lustre-OST0000 complete [ 2818.257719] LustreError: 100373:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2828.166944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2828.368748] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2828.375568] LustreError: 98278:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2828.401168] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2828.411620] LustreError: 100840:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2828.421343] LustreError: 100840:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2828.427484] LustreError: 100840:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2828.483467] Lustre: server umount lustre-OST0000 complete [ 2828.485847] LustreError: 100840:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2839.865352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2840.122330] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2840.127937] Lustre: Found index 0 for lustre-OST0000, updating log [ 2840.135566] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2840.261246] Lustre: lustre-OST0000: new disk, initializing [ 2840.266629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2840.278542] Lustre: 101374:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2840.636828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.619588] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2853.765153] Lustre: Failing over lustre-OST0000 [ 2853.905554] Lustre: server umount lustre-OST0000 complete [ 2857.952427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.955504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2857.984138] Lustre: Skipped 1 previous similar message [ 2857.999579] Lustre: Skipped 1 previous similar message [ 2863.082866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2863.092551] Lustre: Skipped 1 previous similar message [ 2868.196603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.212441] Lustre: Skipped 1 previous similar message [ 2872.289512] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2872.816183] Lustre: server umount lustre-MDT0000 complete [ 2873.249126] Lustre: 93300:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788839578/real 1788839578] req@00000000b0c9fc86 x1875733966330624/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788839594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2873.291371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2873.333971] LustreError: 99020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.354454] LustreError: 99020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2877.294277] LustreError: 98270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839598 with bad export cookie 16876527971683714903 [ 2877.299988] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.309372] LustreError: 98270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2877.865155] Lustre: server umount lustre-MDT0001 complete [ 2888.747856] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 23:53:28 (1788839608) [ 2907.400205] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2918.559671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2919.039302] LustreError: 103722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.219878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2925.051679] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2935.204814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.090838] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2944.351279] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2951.723507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2952.481613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.487926] Lustre: Skipped 1 previous similar message [ 2952.495904] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2958.806169] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2960.690338] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2960.739679] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2960.740312] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2967.431774] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2967.560542] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2967.643641] Lustre: lustre-OST0001: new disk, initializing [ 2967.648904] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2967.656702] Lustre: 106043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2973.638441] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2975.430959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2975.443059] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2975.523511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2981.783500] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2993.380980] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2997.730457] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2997.742792] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.769581] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3003.728555] Lustre: server umount lustre-MDT0001 complete [ 3006.443833] LustreError: 103719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3006.470576] LustreError: 103719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3013.619699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.752094] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3013.756443] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3013.845973] Lustre: 103711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3015.561463] LustreError: 107581:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3015.573129] LustreError: 107581:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3015.591818] LustreError: 103714:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3015.782076] LustreError: 107591:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.116@tcp: cfg command failed: rc = -17 [ 3015.812105] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3015.812105] [ 3020.266642] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3020.417454] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3030.786817] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 23:55:50 (1788839750) [ 3051.846853] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3074.103853] Lustre: 109659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3097.517323] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3102.292063] Lustre: 110793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3188.194159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3188.204639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.219033] Lustre: Skipped 2 previous similar messages [ 3188.223386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3188.228399] Lustre: Skipped 4 previous similar messages [ 3192.643884] Lustre: server umount lustre-MDT0000 complete [ 3194.351755] LustreError: 103718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3194.369569] LustreError: 103718:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3196.506031] LustreError: 106042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839917 with bad export cookie 16876527971683715904 [ 3196.509109] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3196.518403] LustreError: 106042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3196.965832] Lustre: server umount lustre-MDT0001 complete [ 3211.102316] Lustre: server umount lustre-OST0000 complete [ 3215.329267] Lustre: 93299:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788839920/real 1788839920] req@ffff98040897f100 x1875733966599168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788839936 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3215.367528] Lustre: 93299:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3215.376786] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.395382] Lustre: Skipped 3 previous similar messages [ 3216.078363] Lustre: server umount lustre-OST0001 complete [ 3221.538621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3232.223944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3241.581052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3252.722485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3273.774987] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3285.391971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.408449] Lustre: lustre-MDT0000: reset Object Index mappings [ 3285.954796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3285.960713] Lustre: Skipped 2 previous similar messages [ 3290.625743] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3299.164230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3299.185333] Lustre: lustre-MDT0001: reset Object Index mappings [ 3304.473710] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3307.550824] Lustre: 115894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3314.819628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3322.492666] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3324.967565] LustreError: 116248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3324.992674] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3325.004275] LustreError: 116248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3331.068584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 3331.447355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3337.761040] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3337.770015] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3338.781917] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3346.985934] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3350.631879] Lustre: 117784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3400.950292] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:02:01 (1788840121) [ 3429.867820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.877312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3429.893082] Lustre: Skipped 2 previous similar messages [ 3429.902678] Lustre: Skipped 7 previous similar messages [ 3434.805969] Lustre: server umount lustre-MDT0000 complete [ 3439.138316] LustreError: 114728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840160 with bad export cookie 16876527971683762755 [ 3439.166247] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3445.904962] Lustre: server umount lustre-MDT0001 complete [ 3456.989116] Lustre: server umount lustre-OST0000 complete [ 3467.386697] Lustre: server umount lustre-OST0001 complete [ 3477.171331] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 3491.894394] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3531.093730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3531.460563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3531.513345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3531.627887] Lustre: lustre-MDT0000: new disk, initializing [ 3531.706498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3531.710675] Lustre: Skipped 3 previous similar messages [ 3531.727826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3537.205965] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3544.755674] Lustre: server umount lustre-MDT0000 complete [ 3555.517348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.820449] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3555.832452] Lustre: 122066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3555.843924] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3560.310982] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3565.421811] Lustre: server umount lustre-MDT0000 complete [ 3573.988710] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:04:54 (1788840294) [ 3575.993360] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3577.986894] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:04:58 (1788840298) [ 3596.375741] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3606.619752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3611.810326] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3620.815132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.927610] Lustre: 124118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3621.041182] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3621.051068] Lustre: Skipped 1 previous similar message [ 3621.224806] Lustre: lustre-MDT0001: new disk, initializing [ 3621.379064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3621.392555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3626.550740] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3632.442768] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3641.498873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3641.876871] Lustre: lustre-OST0000: new disk, initializing [ 3641.883936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3641.890910] Lustre: 125697:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3643.518930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3643.527324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3643.743587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3650.546560] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3663.516936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3663.681102] Lustre: lustre-OST0001: new disk, initializing [ 3663.686796] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3663.693035] Lustre: 126569:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3665.736081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3665.752084] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3665.895510] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3671.691449] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3682.693115] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3687.016274] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4094.496406] Lustre: 132985:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4094.509428] Lustre: 132985:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4094.515866] Lustre: 132985:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4094.525118] Lustre: 132985:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4094.536844] Lustre: 132985:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4094.549962] Lustre: 132985:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4392.743652] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:18:32 (1788841112) [ 4408.290637] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4408.301037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4408.307846] Lustre: Skipped 4 previous similar messages [ 4408.315546] Lustre: Skipped 4 previous similar messages [ 4414.374417] Lustre: server umount lustre-MDT0000 complete [ 4418.540580] LustreError: 127406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4418.583636] LustreError: 127406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 4418.764234] LustreError: 125695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841139 with bad export cookie 16876527971683831285 [ 4418.771490] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4418.781213] LustreError: 125695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4423.652923] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4423.659506] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4423.676722] Lustre: Skipped 3 previous similar messages [ 4423.688525] Lustre: Skipped 8 previous similar messages [ 4433.375183] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4433.883343] Lustre: server umount lustre-MDT0001 complete [ 4451.807575] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4451.915838] Lustre: server umount lustre-OST0000 complete [ 4462.419907] Lustre: server umount lustre-OST0001 complete [ 4469.884820] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 4480.119332] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 4520.622864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4520.976789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4521.072318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4521.154300] Lustre: lustre-MDT0000: new disk, initializing [ 4521.291781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4521.299815] Lustre: Skipped 5 previous similar messages [ 4521.334334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4527.269533] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4539.453267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4539.571775] Lustre: 136048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4539.624071] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4539.627924] Lustre: Skipped 1 previous similar message [ 4539.704666] Lustre: lustre-MDT0001: new disk, initializing [ 4539.872575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4539.889975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4545.645245] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4557.852916] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4566.679086] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4575.831373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4576.468852] Lustre: lustre-OST0000: new disk, initializing [ 4576.487683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4576.511507] Lustre: 138017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4576.685732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4576.699578] Lustre: Skipped 1 previous similar message [ 4577.939660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4577.953451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4578.216768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4584.234690] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4594.097832] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4604.580298] Lustre: lustre-MDD0000: changelog on [ 4607.221685] Lustre: lustre-MDD0001: changelog on [ 6444.982677] Lustre: lustre-MDD0001: changelog off [ 6447.134927] Lustre: 137160:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6448.146940] Lustre: lustre-MDD0000: changelog off [ 6457.317724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6457.325665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6457.333538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6457.344209] Lustre: Skipped 1 previous similar message [ 6457.368489] Lustre: Skipped 5 previous similar messages [ 6461.429564] Lustre: server umount lustre-OST0000 complete [ 6467.552249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.555141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6467.596166] Lustre: Skipped 1 previous similar message [ 6467.620190] Lustre: Skipped 1 previous similar message [ 6472.677371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6472.693219] Lustre: Skipped 1 previous similar message [ 6480.351622] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6480.789392] Lustre: server umount lustre-MDT0000 complete [ 6482.914144] LustreError: 136058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.959443] LustreError: 136058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6485.708990] LustreError: 136039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843206 with bad export cookie 16876527971684650880 [ 6485.727728] LustreError: 136039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6485.733354] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6486.032969] Lustre: server umount lustre-MDT0001 complete [ 6500.911668] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 6505.199084] Key type lgssc unregistered [ 6505.650995] LNet: 147146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6505.685426] LNetError: 147146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6505.731669] LNet: Removed LNI 192.168.204.116@tcp [ 6507.579825] Key type .llcrypt unregistered [ 6507.581636] Key type ._llcrypt unregistered [ 6530.807917] Key type ._llcrypt registered [ 6530.812256] Key type .llcrypt registered [ 6530.934795] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 6549.134344] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 6550.853832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6551.052840] alg: No test for adler32 (adler32-zlib) [ 6552.167143] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 6552.439328] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 6554.135221] Key type lgssc registered [ 6556.010762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6599.500596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6599.543153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6600.841624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6600.881204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6600.996836] Lustre: lustre-MDT0000: new disk, initializing [ 6601.152736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6601.201539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6608.169599] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6621.205470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6621.359796] Lustre: 150365:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6621.412666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6621.415887] Lustre: Skipped 1 previous similar message [ 6621.474297] Lustre: lustre-MDT0001: new disk, initializing [ 6621.545436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6621.575647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6621.593122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6627.685783] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6638.009593] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6643.401326] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6650.994322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6651.275152] Lustre: lustre-OST0000: new disk, initializing [ 6651.280760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6651.290293] Lustre: 152336:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6651.401122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6657.832115] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6660.675600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6660.685594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6660.894389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6668.209405] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6673.758456] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_module obdecho/obdecho [ 6675.649378] LustreError: 153337:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6675.659892] LustreError: 153337:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6677.525037] LustreError: 153433:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6691.816162] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6691.836519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6691.837915] Lustre: Skipped 2 previous similar messages [ 6691.848360] Lustre: Skipped 1 previous similar message [ 6694.888634] Lustre: server umount lustre-MDT0000 complete [ 6698.540367] LustreError: 150356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843419 with bad export cookie 18231279663760769479 [ 6698.544084] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6698.552214] LustreError: 150356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6698.951959] Lustre: server umount lustre-MDT0001 complete [ 6713.762803] Lustre: server umount lustre-OST0000 complete [ 6732.985754] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 6744.505456] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 6782.274944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.609182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6782.669520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6782.809850] Lustre: lustre-MDT0000: new disk, initializing [ 6782.940518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6782.967515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6787.749992] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6798.946747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6799.065504] Lustre: 157149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6799.118878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6799.123203] Lustre: Skipped 1 previous similar message [ 6799.184932] Lustre: lustre-MDT0001: new disk, initializing [ 6799.271042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6799.311701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6799.322248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6803.993927] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6813.173707] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6819.216872] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6827.810597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6828.127414] Lustre: lustre-OST0000: new disk, initializing [ 6828.133392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6828.144882] Lustre: 159117:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6828.312794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6835.767990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6835.782907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6835.946090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6836.872801] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6847.766550] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6853.217764] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6853.519510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6859.051758] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6859.333787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6861.800875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6861.804273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6861.814966] LustreError: Skipped 1 previous similar message [ 6861.815618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6861.824809] Lustre: Skipped 1 previous similar message [ 6861.843878] Lustre: Skipped 2 previous similar messages [ 6866.916916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6866.929701] Lustre: Skipped 1 previous similar message [ 6867.773363] Lustre: server umount lustre-OST0000 complete [ 6876.131470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6876.153991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6876.172968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6878.822996] Lustre: server umount lustre-MDT0000 complete [ 6882.273728] LustreError: 158309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6882.295839] LustreError: 158309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6882.751805] LustreError: 157140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843603 with bad export cookie 18231279663760770438 [ 6882.753991] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6882.764255] LustreError: 157140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6883.084776] Lustre: server umount lustre-MDT0001 complete [ 6887.221970] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:00:07 (1788843607) [ 6888.952417] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6890.830091] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 01:00:11 (1788843611) [ 6899.240350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6899.804052] LustreError: 161531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6899.926516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6905.001226] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6905.320108] LustreError: 161532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6909.414094] LustreError: 161531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6914.533922] LustreError: 161532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6917.013385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6925.465476] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6933.809753] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6940.295531] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6947.080894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6947.520540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6947.530929] Lustre: Skipped 1 previous similar message [ 6955.126929] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6965.302636] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6979.047987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6979.057830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6979.085379] Lustre: Skipped 1 previous similar message [ 6979.091495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6979.100858] Lustre: Skipped 2 previous similar messages [ 6983.610326] Lustre: server umount lustre-MDT0000 complete [ 6985.702075] LustreError: 161532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6985.722804] LustreError: 161532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6987.788048] LustreError: 161512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843708 with bad export cookie 18231279663760771320 [ 6987.825498] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6990.823813] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6990.840942] Lustre: Skipped 2 previous similar messages [ 6990.844551] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6990.849993] Lustre: Skipped 4 previous similar messages [ 6994.330408] Lustre: server umount lustre-MDT0001 complete [ 7001.938434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7015.467615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7015.638931] LustreError: 165410:0:(llog.c:1655:llog_backup()) MGC192.168.204.116@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7015.645220] Lustre: 165410:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.116@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7017.441503] LustreError: 165415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7017.458722] LustreError: 165415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7053.226883] LustreError: 165415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7053.265581] LustreError: 165415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 7076.319887] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7076.325269] Lustre: 165410:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7117.310591] LustreError: 165432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7117.312049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7117.343727] LustreError: 165432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 7122.433594] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7123.907723] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7135.079108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7138.789591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7158.250162] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfd028442bf43a251 [ 7158.266156] Lustre: MGC192.168.204.116@tcp: Connection restored to 0@lo (at 0@lo) [ 7163.906288] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7164.788682] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7174.622947] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7176.682229] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7184.442657] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7187.066350] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7209.957283] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7209.980099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7209.984249] Lustre: Skipped 1 previous similar message [ 7209.984820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7210.021115] Lustre: Skipped 4 previous similar messages [ 7214.754692] Lustre: server umount lustre-MDT0000 complete [ 7219.206536] LustreError: 161511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843940 with bad export cookie 18231279663760777809 [ 7219.229904] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7219.232564] LustreError: 161511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7219.805546] Lustre: server umount lustre-MDT0001 complete [ 7234.491293] Lustre: server umount lustre-OST0000 complete [ 7259.010707] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 7272.237715] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 7314.249132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7314.464911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7314.485172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7314.546455] Lustre: lustre-MDT0000: new disk, initializing [ 7314.615469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7314.625719] Lustre: Skipped 1 previous similar message [ 7314.639747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7319.738447] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7331.795116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7331.909746] Lustre: 170526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7332.282504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7337.745795] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7347.754628] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7353.805989] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7361.155075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7361.394776] Lustre: lustre-OST0000: new disk, initializing [ 7361.398588] Lustre: Skipped 1 previous similar message [ 7361.403066] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7361.407800] Lustre: Skipped 2 previous similar messages [ 7361.411996] Lustre: 172494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7369.007643] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7371.324592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7371.344889] Lustre: Skipped 1 previous similar message [ 7371.354787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7371.534708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7381.059462] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7387.359966] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7387.750514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7393.563385] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7393.827983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7397.351455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7397.353199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7397.361275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7397.370798] Lustre: Skipped 2 previous similar messages [ 7401.982278] Lustre: server umount lustre-OST0000 complete [ 7412.708701] LustreError: 173105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7412.727677] LustreError: 173105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 7412.803044] Lustre: server umount lustre-MDT0000 complete [ 7417.290313] LustreError: 170517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844138 with bad export cookie 18231279663760794623 [ 7417.310900] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7417.312933] LustreError: 170517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7417.885767] Lustre: server umount lustre-MDT0001 complete [ 7427.760572] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 01:09:07 (1788844147) [ 7435.525685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7441.306957] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7451.458137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7457.751285] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7466.196936] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7472.503777] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7480.272701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7487.755477] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7495.635716] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7505.475181] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 01:10:25 (1788844225) [ 7511.533649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7511.534929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7511.553473] Lustre: Skipped 2 previous similar messages [ 7511.562485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7511.563670] LustreError: Skipped 1 previous similar message [ 7511.586550] Lustre: Skipped 4 previous similar messages [ 7515.354653] Lustre: server umount lustre-OST0000 complete [ 7525.751077] Lustre: server umount lustre-MDT0000 complete [ 7530.057343] LustreError: 174888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844250 with bad export cookie 18231279663760795505 [ 7530.070613] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7530.073527] LustreError: 174888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7530.617294] Lustre: server umount lustre-MDT0001 complete [ 7543.407142] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 7547.089219] Key type lgssc unregistered [ 7547.562708] LNet: 178633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7547.570550] LNetError: 178633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7547.590906] LNet: Removed LNI 192.168.204.116@tcp [ 7548.945996] Key type .llcrypt unregistered [ 7548.948736] Key type ._llcrypt unregistered [ 7571.920483] Key type ._llcrypt registered [ 7571.922952] Key type .llcrypt registered [ 7572.029781] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 7573.416504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7573.624919] alg: No test for adler32 (adler32-zlib) [ 7574.669767] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 7574.996290] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 7576.735264] Key type lgssc registered [ 7577.979158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7588.757680] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7588.793968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7604.513524] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7609.632854] LustreError: 179876:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 7614.751554] LustreError: 179876:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7614.792090] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7614.811252] LustreError: MGC192.168.204.116@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7614.832656] LustreError: 179876:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7614.849618] LustreError: 179876:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7614.863635] LustreError: 179876:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7615.002643] Lustre: server umount lustre-OST0000 complete [ 7615.011192] LustreError: 179876:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7625.099203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7626.538422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7631.518648] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7636.973188] LustreError: 180377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7637.005130] LustreError: 180377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7641.768726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7642.080069] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7642.415327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7647.515118] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7655.230906] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7661.600175] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7668.852419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7669.085250] LustreError: 182192:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7669.095919] Lustre: 182192:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.116@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7669.261709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.477122] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7684.938614] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7688.673409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7688.678726] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7688.699856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7688.742600] Lustre: Skipped 1 previous similar message [ 7693.799272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7693.808802] Lustre: Skipped 4 previous similar messages [ 7694.098341] Lustre: server umount lustre-MDT0000 complete [ 7698.599905] LustreError: 180355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844419 with bad export cookie 17331309495515662883 [ 7698.611034] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7698.915776] LustreError: 180371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7698.917543] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7698.943367] LustreError: 181105:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000af5454d0 x1875739195504384/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7698.953284] Lustre: Skipped 1 previous similar message [ 7698.958497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7699.101923] Lustre: server umount lustre-MDT0001 complete [ 7706.435533] Lustre: server umount lustre-OST0000 complete [ 7719.470091] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 7732.382713] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 7780.874658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7781.224464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7781.293941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7781.464294] Lustre: lustre-MDT0000: new disk, initializing [ 7781.580654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7781.598534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7787.279702] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7799.636645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7799.886995] Lustre: 185720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7799.974102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7799.994725] Lustre: Skipped 1 previous similar message [ 7800.219844] Lustre: lustre-MDT0001: new disk, initializing [ 7800.351021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7800.391831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7800.412948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7805.679834] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7817.921925] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7824.614986] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7832.074748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7832.482362] Lustre: lustre-OST0000: new disk, initializing [ 7832.487317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7832.502272] Lustre: 187689:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7832.611124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7839.806354] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7841.867349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7841.879352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7842.042619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7850.351927] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7856.153809] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7856.384293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7861.438315] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7861.693603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7867.873174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7867.885371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7867.888710] LustreError: Skipped 1 previous similar message [ 7867.890199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7867.934803] Lustre: Skipped 1 previous similar message [ 7870.632461] Lustre: server umount lustre-OST0000 complete [ 7877.088594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7877.108347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7877.127823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7877.139148] Lustre: Skipped 1 previous similar message [ 7890.399383] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7890.736498] Lustre: server umount lustre-MDT0000 complete [ 7892.449034] LustreError: 185732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7892.465170] LustreError: 185732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7894.785337] LustreError: 185712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844615 with bad export cookie 17331309495515663604 [ 7894.798728] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7894.808924] LustreError: 185712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7895.152828] Lustre: server umount lustre-MDT0001 complete [ 7904.983091] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 01:17:04 (1788844624) [ 7924.048861] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 7927.562950] Key type lgssc unregistered [ 7927.956403] LNet: 190636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7927.968129] LNetError: 190636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7928.006201] LNet: Removed LNI 192.168.204.116@tcp [ 7929.111392] Key type .llcrypt unregistered [ 7929.114122] Key type ._llcrypt unregistered [ 7956.083332] Key type ._llcrypt registered [ 7956.090357] Key type .llcrypt registered [ 7956.318030] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 7957.632423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7957.811377] alg: No test for adler32 (adler32-zlib) [ 7959.012271] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 7959.348462] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 7961.135184] Key type lgssc registered [ 7962.755848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7971.735402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7971.763830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7973.357589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7978.314901] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7983.588481] LustreError: 191854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7983.615902] LustreError: 191854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7988.293749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.678814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7994.222644] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8005.573809] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8012.911338] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8025.136478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8025.471478] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8025.475501] LustreError: 193718:0:(llog.c:1696:llog_backup()) MGC192.168.204.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8025.497850] Lustre: 193718:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.116@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8025.733561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8033.951390] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8042.565706] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8048.611322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8048.615603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8048.640070] Lustre: Skipped 2 previous similar messages [ 8048.673203] Lustre: Skipped 2 previous similar messages [ 8051.759021] Lustre: server umount lustre-MDT0000 complete [ 8053.740894] LustreError: 191854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8053.755670] LustreError: 191854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8056.221289] LustreError: 191836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844777 with bad export cookie 6511570065637281738 [ 8056.235149] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8056.241432] LustreError: 191836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8056.741807] Lustre: server umount lustre-MDT0001 complete [ 8071.866861] Lustre: server umount lustre-OST0000 complete [ 8083.941991] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 8096.256652] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 8146.242679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8146.496290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8146.541743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8146.617791] Lustre: lustre-MDT0000: new disk, initializing [ 8146.739363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8146.762898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8152.026164] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8162.911625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8163.063269] Lustre: 197246:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8163.126637] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8163.130073] Lustre: Skipped 1 previous similar message [ 8163.133715] LustreError: 198031:0:(llog.c:1696:llog_backup()) MGC192.168.204.116@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8163.146438] LustreError: 198031:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8163.155350] Lustre: 198031:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.116@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8163.195810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8163.204389] Lustre: Skipped 1 previous similar message [ 8163.322966] Lustre: lustre-MDT0001: new disk, initializing [ 8163.490773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8163.574604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8163.585397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8168.954101] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8178.557248] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8185.123875] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8192.097877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8192.356380] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8192.362150] Lustre: Skipped 1 previous similar message [ 8192.365712] LustreError: 199193:0:(llog.c:1696:llog_backup()) MGC192.168.204.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8192.378614] LustreError: 199193:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8192.383914] Lustre: 199193:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.116@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8192.396805] Lustre: 199193:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8192.417465] Lustre: lustre-OST0000: new disk, initializing [ 8192.426122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8192.436654] Lustre: 199216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8192.537522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8198.717712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8198.741227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8198.871660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8199.703446] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8210.090841] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8215.093915] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8215.314473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8220.277424] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8220.482278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8224.736225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8224.737389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8224.750282] LustreError: Skipped 1 previous similar message [ 8224.754081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8224.769964] Lustre: Skipped 1 previous similar message [ 8228.841544] Lustre: server umount lustre-OST0000 complete [ 8234.976827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8234.978643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8234.995699] Lustre: Skipped 1 previous similar message [ 8235.029658] Lustre: Skipped 2 previous similar messages [ 8240.098740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8240.122337] Lustre: Skipped 1 previous similar message [ 8244.197119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8244.211352] Lustre: Skipped 1 previous similar message [ 8248.287693] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8248.560947] Lustre: server umount lustre-MDT0000 complete [ 8249.316307] LustreError: 197255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8249.353517] LustreError: 197255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8253.532710] LustreError: 197238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844974 with bad export cookie 6511570065637282459 [ 8253.558530] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8253.572249] LustreError: 197238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8254.203589] Lustre: server umount lustre-MDT0001 complete [ 8263.907447] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 01:23:04 (1788844984) [ 8280.984966] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 8293.249771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8293.776818] LustreError: 202348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8293.870467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8298.886263] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8298.977744] LustreError: 202349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8308.192101] LustreError: 202348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8308.209704] LustreError: 202348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8308.280367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8313.537899] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8316.530303] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8323.162835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8330.203428] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8340.340708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8340.539320] Lustre: lustre-OST0001: new disk, initializing [ 8340.542674] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8340.548244] Lustre: 204655:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8340.644867] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8340.652969] Lustre: Skipped 2 previous similar messages [ 8348.920517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8348.942464] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8349.082511] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8349.158842] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8361.272968] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8368.975369] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8371.239739] Lustre: DEBUG MARKER: ADD OST3 [ 8377.700524] loop: module loaded [ 8377.799389] loop0: detected capacity change from 0 to 204800000 [ 8378.041053] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8378.066092] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8378.078190] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8385.494861] loop0: detected capacity change from 0 to 204800000 [ 8385.875402] Lustre: DEBUG MARKER: STOP OST3 [ 8393.126209] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8393.522479] Lustre: lustre-OST0003: new disk, initializing [ 8393.529915] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8393.545878] Lustre: 206701:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8395.618762] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8395.790881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8395.813743] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8395.875144] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8397.199964] Lustre: Failing over lustre-OST0003 [ 8397.322690] Lustre: server umount lustre-OST0003 complete [ 8397.667201] LustreError: 206839:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8400.876059] LustreError: 203833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8400.879773] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8400.902572] LustreError: 203833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8400.937742] Lustre: Skipped 1 previous similar message [ 8402.455170] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8414.229387] Lustre: DEBUG MARKER: START OST3 again [ 8421.612830] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8422.007456] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8422.015523] Lustre: Skipped 1 previous similar message [ 8422.035506] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8423.548631] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8423.625121] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8423.625179] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8428.880769] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8450.645185] Lustre: Failing over lustre-OST0003 [ 8450.904356] Lustre: server umount lustre-OST0003 complete [ 8453.103643] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8453.108495] LustreError: 203835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8453.115404] Lustre: Skipped 1 previous similar message [ 8453.129799] LustreError: 203835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 8472.544562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8472.554551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8472.569434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8475.868299] Lustre: server umount lustre-MDT0000 complete [ 8480.010423] LustreError: 203856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845200 with bad export cookie 6511570065637283341 [ 8480.033113] LustreError: 203856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8480.058660] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8486.515240] Lustre: server umount lustre-MDT0001 complete [ 8490.999443] Lustre: server umount lustre-OST0000 complete [ 8496.035986] Lustre: server umount lustre-OST0001 complete [ 8503.365769] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 8512.645484] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 8549.395024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8549.656190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8549.695799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8549.755246] Lustre: lustre-MDT0000: new disk, initializing [ 8549.816576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8554.743654] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8566.158154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8566.394524] Lustre: 212268:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8566.464503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8566.468211] Lustre: Skipped 1 previous similar message [ 8566.615379] Lustre: lustre-MDT0001: new disk, initializing [ 8566.762301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8566.774320] Lustre: Skipped 1 previous similar message [ 8566.796629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8566.814542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8571.526351] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8581.788906] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8588.432609] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8595.719622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8595.990865] Lustre: 214236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8597.396675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8597.532912] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8602.483562] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8612.801816] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8616.664073] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8616.961517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8622.465691] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8622.760307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8628.198156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8628.200911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8628.211994] LustreError: Skipped 1 previous similar message [ 8628.224754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8628.224763] Lustre: Skipped 8 previous similar messages [ 8628.276026] Lustre: Skipped 6 previous similar messages [ 8631.476598] Lustre: server umount lustre-OST0000 complete [ 8638.440243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8641.794919] Lustre: server umount lustre-MDT0000 complete [ 8643.555219] LustreError: 212279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8643.573965] LustreError: 212279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 8645.987253] LustreError: 212258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845366 with bad export cookie 6511570065637285637 [ 8645.998785] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8646.002817] LustreError: 212258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8646.771356] Lustre: server umount lustre-MDT0001 complete [ 8650.720286] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 01:29:30 (1788845370) [ 8667.615312] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 8679.186487] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 8715.152414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8715.398244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8715.435426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8715.439206] Lustre: Skipped 1 previous similar message [ 8715.520765] Lustre: lustre-MDT0000: new disk, initializing [ 8715.525520] Lustre: Skipped 1 previous similar message [ 8715.656528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8715.665660] Lustre: Skipped 1 previous similar message [ 8720.351464] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8731.437559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8731.567340] Lustre: 218844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8731.817080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8736.970787] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8747.394117] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8753.866479] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8761.661230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8761.881519] Lustre: 220813:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8763.553886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8769.120714] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8779.699654] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8785.208937] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8785.463539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8791.104934] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8791.307947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8794.086396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8794.101642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.119197] Lustre: Skipped 2 previous similar messages [ 8794.124771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8794.132065] Lustre: Skipped 3 previous similar messages [ 8799.884569] Lustre: server umount lustre-OST0000 complete [ 8803.813526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8803.836348] LustreError: Skipped 1 previous similar message [ 8809.548676] Lustre: server umount lustre-MDT0000 complete [ 8809.958561] LustreError: 218851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8809.982307] LustreError: 218851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8812.843773] LustreError: 218836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845533 with bad export cookie 6511570065637286519 [ 8812.844452] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8812.853512] LustreError: 218836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8813.267552] Lustre: server umount lustre-MDT0001 complete [ 8821.558590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8826.121140] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8834.426801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8834.856312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8834.866466] Lustre: Skipped 5 previous similar messages [ 8838.862913] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8846.168483] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8851.373381] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8858.096453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8864.345852] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8871.385691] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8882.145066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8882.157308] Lustre: Skipped 3 previous similar messages [ 8882.159967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8882.175341] Lustre: Skipped 7 previous similar messages [ 8887.340352] Lustre: server umount lustre-OST0000 complete [ 8906.207155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8906.533885] Lustre: server umount lustre-MDT0000 complete [ 8911.110143] LustreError: 223011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845632 with bad export cookie 6511570065637287401 [ 8911.115687] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8911.123761] LustreError: 223011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8911.637699] Lustre: server umount lustre-MDT0001 complete [ 9010.161502] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 9014.145575] Key type lgssc unregistered [ 9014.618287] LNet: 226692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9014.626664] LNetError: 226692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9015.654910] LNet: Removed LNI 192.168.204.116@tcp [ 9017.318427] Key type .llcrypt unregistered [ 9017.320927] Key type ._llcrypt unregistered [ 9046.601973] Key type ._llcrypt registered [ 9046.606880] Key type .llcrypt registered [ 9046.730031] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 9063.249265] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 9065.058851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9065.099798] alg: No test for adler32 (adler32-zlib) [ 9066.200474] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 9066.551241] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 9068.271290] Key type lgssc registered [ 9069.652294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9108.977859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9108.992754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9110.246564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9110.306447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9110.489760] Lustre: lustre-MDT0000: new disk, initializing [ 9110.687274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9110.728281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9116.330427] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9128.855953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9129.065398] Lustre: 230042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9129.180365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9129.187380] Lustre: Skipped 1 previous similar message [ 9129.311331] Lustre: lustre-MDT0001: new disk, initializing [ 9129.423545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9129.491929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9129.504253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9134.731146] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9144.323255] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9149.660285] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9157.272707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9157.626531] Lustre: lustre-OST0000: new disk, initializing [ 9157.633906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9157.640840] Lustre: 232012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9157.752663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9163.441804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9163.453189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9163.761076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9165.625770] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9176.382876] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9181.720312] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9181.947298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9186.985933] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9187.218541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9189.344393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9189.354318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9189.365261] LustreError: Skipped 1 previous similar message [ 9189.374207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9189.385444] Lustre: Skipped 1 previous similar message [ 9194.471503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9194.479132] Lustre: Skipped 1 previous similar message [ 9195.433289] Lustre: server umount lustre-OST0000 complete [ 9201.124646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9201.143513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9201.159805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9201.170740] Lustre: Skipped 1 previous similar message [ 9204.709419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9204.716563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9206.538559] Lustre: server umount lustre-MDT0000 complete [ 9211.304397] LustreError: 230034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845932 with bad export cookie 321709393927360754 [ 9211.317395] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9211.323020] LustreError: 230034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9211.787230] Lustre: server umount lustre-MDT0001 complete [ 9236.643132] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 9248.494884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9248.913302] LustreError: 234963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9248.940456] LustreError: 234963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9249.021769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9254.055371] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9254.391668] LustreError: 234964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9254.428787] LustreError: 234964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9258.467088] LustreError: 234963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9262.863223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9263.198984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9267.739521] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9270.481522] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9278.647206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9279.158987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9286.265088] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9294.253799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9294.593527] Lustre: lustre-OST0001: new disk, initializing [ 9294.598579] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9294.606443] Lustre: 237270:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9300.101104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9300.109127] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9300.226755] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9300.796165] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9312.288452] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9317.744103] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9335.210598] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 01:40:54 (1788846054) [ 9339.873238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9339.887674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9339.912384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9339.925971] Lustre: Skipped 1 previous similar message [ 9344.000165] Lustre: server umount lustre-MDT0000 complete [ 9346.540191] LustreError: 234959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9346.579722] LustreError: 234959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9348.719544] LustreError: 237269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846069 with bad export cookie 321709393927361636 [ 9348.722870] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9348.745300] LustreError: 237269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9349.185551] Lustre: server umount lustre-MDT0001 complete [ 9364.873250] Lustre: server umount lustre-OST0000 complete [ 9368.031222] Lustre: 228265:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788846072/real 1788846072] req@ffff980303742300 x1875740760064256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788846088 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9368.060548] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9368.071645] Lustre: Skipped 3 previous similar messages [ 9368.711752] Lustre: server umount lustre-OST0001 complete [ 9375.001093] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 9386.254163] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 9421.718081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9421.978596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9422.004784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9422.082721] Lustre: lustre-MDT0000: new disk, initializing [ 9422.184468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9422.190825] Lustre: Skipped 1 previous similar message [ 9422.201929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9426.475845] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9437.145093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9437.343326] Lustre: 241217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9437.443835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9437.448730] Lustre: Skipped 1 previous similar message [ 9437.596646] Lustre: lustre-MDT0001: new disk, initializing [ 9437.749121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9437.764431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9442.275865] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9450.719404] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9456.441405] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9464.052234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9464.512692] Lustre: lustre-OST0000: new disk, initializing [ 9464.520039] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9464.529090] Lustre: 243184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9464.635494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9464.646302] Lustre: Skipped 1 previous similar message [ 9466.506642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9466.521141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9466.618857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9470.660930] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9480.914772] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9485.806740] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9486.089716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9492.362610] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9492.645627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9497.570434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9497.573597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9497.594839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9497.622327] Lustre: Skipped 5 previous similar messages [ 9509.343118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9509.522225] Lustre: server umount lustre-OST0000 complete [ 9514.468911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9514.486136] LustreError: Skipped 1 previous similar message [ 9514.491164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9514.510865] Lustre: Skipped 1 previous similar message [ 9514.528293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9514.534792] Lustre: Skipped 4 previous similar messages [ 9528.287132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9528.710488] Lustre: server umount lustre-MDT0000 complete [ 9533.131171] LustreError: 241210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846254 with bad export cookie 321709393927363771 [ 9533.135329] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9533.143734] LustreError: 241210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9533.705734] Lustre: server umount lustre-MDT0001 complete [ 9544.022808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9544.601359] LustreError: 245405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9544.706112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9550.363558] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9554.915281] LustreError: 245406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9554.944704] LustreError: 245406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9560.609147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9566.592328] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9574.550340] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9580.615416] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9587.525911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9595.233559] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9604.066840] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9749.987302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9749.993854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9750.015197] Lustre: Skipped 1 previous similar message [ 9750.024143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9750.032177] Lustre: Skipped 6 previous similar messages [ 9755.508625] Lustre: server umount lustre-OST0000 complete [ 9765.701875] Lustre: server umount lustre-MDT0000 complete [ 9769.442845] LustreError: 246140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9769.491568] LustreError: 246140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9770.005203] LustreError: 245386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846490 with bad export cookie 321709393927364653 [ 9770.006467] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9770.014271] LustreError: 245386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9770.542254] Lustre: server umount lustre-MDT0001 complete [ 9876.160506] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 9880.379768] Key type lgssc unregistered [ 9880.902857] LNet: 249011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9880.918801] LNetError: 249011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9880.951420] LNet: Removed LNI 192.168.204.116@tcp [ 9882.449137] Key type .llcrypt unregistered [ 9882.450901] Key type ._llcrypt unregistered [ 9917.938513] Key type ._llcrypt registered [ 9917.941721] Key type .llcrypt registered [ 9918.051703] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 9935.684232] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 9936.868119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9937.049841] alg: No test for adler32 (adler32-zlib) [ 9938.214873] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 9938.502257] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 9940.175178] Key type lgssc registered [ 9941.427282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9997.332571] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [10011.104897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10011.130296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10012.324105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10012.358614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10012.437236] Lustre: lustre-MDT0000: new disk, initializing [10012.499582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10012.518717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10018.690872] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10031.447437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10031.695852] Lustre: 253078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10031.813906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10031.821165] Lustre: Skipped 1 previous similar message [10031.950790] Lustre: lustre-MDT0001: new disk, initializing [10032.146279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10032.209683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10032.224994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10037.209455] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10042.268168] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10049.360926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10049.658606] Lustre: lustre-OST0000: new disk, initializing [10049.661758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10049.669968] Lustre: 254713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10049.752480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10053.745579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10053.766390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10053.938327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10057.221859] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10069.282257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10069.518871] Lustre: lustre-OST0001: new disk, initializing [10069.526478] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10069.535337] Lustre: 255586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10069.665224] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10077.377346] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10079.856554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10079.867576] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10079.979849] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10088.417729] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10094.549560] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10108.906531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10108.920087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10108.950666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10110.944179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10110.948150] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10110.965930] Lustre: Skipped 2 previous similar messages [10113.465893] Lustre: server umount lustre-MDT0000 complete [10117.068385] LustreError: 255585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846838 with bad export cookie 12679908560699792966 [10117.076207] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10117.083180] LustreError: 255585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10117.476802] Lustre: server umount lustre-MDT0001 complete [10132.585517] Lustre: server umount lustre-OST0000 complete [10147.050408] Lustre: server umount lustre-OST0001 complete [10163.360923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10175.807920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10185.632973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10196.909245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10217.889181] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [10230.060355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10230.078839] Lustre: lustre-MDT0000: reset Object Index mappings [10230.607432] LustreError: 260598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10230.627527] LustreError: 260598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10230.676244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10235.872104] LustreError: 260599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10236.389629] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10240.994513] LustreError: 260598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10246.123574] LustreError: 260599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10247.299794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10247.351545] Lustre: lustre-MDT0001: reset Object Index mappings [10248.116440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10252.776508] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10256.517392] Lustre: 261741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10264.562184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10265.088260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10272.259906] LustreError: 262097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10273.333371] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10275.384820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10280.441712] LustreError: 262098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10280.479558] LustreError: 262098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10284.868522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10290.191940] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10292.839721] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10301.562812] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10305.453064] Lustre: 263613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10314.720305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10314.731211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10314.753828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10314.760452] Lustre: Skipped 3 previous similar messages [10317.503585] Lustre: server umount lustre-MDT0000 complete [10320.871433] LustreError: 260595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10320.905281] LustreError: 260595:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10321.810623] LustreError: 260578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847042 with bad export cookie 12679908560699799756 [10321.830022] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10322.166138] Lustre: server umount lustre-MDT0001 complete [10336.739560] Lustre: server umount lustre-OST0000 complete [10351.638145] Lustre: server umount lustre-OST0001 complete [10376.282692] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 01:58:16 (1788847096) [10397.908951] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [10412.612712] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [10457.328586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10457.605523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10457.629589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10457.734939] Lustre: lustre-MDT0000: new disk, initializing [10457.843404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10457.848857] Lustre: Skipped 1 previous similar message [10457.877831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10464.153289] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10476.630541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10476.798067] Lustre: 267891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10476.862518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10476.866351] Lustre: Skipped 1 previous similar message [10476.940109] Lustre: lustre-MDT0001: new disk, initializing [10477.150467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10477.162113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10481.706653] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10493.892533] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10500.909120] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10509.347926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10509.746966] Lustre: lustre-OST0000: new disk, initializing [10509.757965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10509.778804] Lustre: 269860:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10511.449874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10511.466803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10511.622434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10517.582558] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10528.584801] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10535.411691] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10535.712971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10542.588211] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10542.990041] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10547.680923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10547.699480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10547.704716] LustreError: Skipped 1 previous similar message [10547.717179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10547.762561] Lustre: Skipped 4 previous similar messages [10547.765537] Lustre: Skipped 4 previous similar messages [10551.621933] Lustre: server umount lustre-OST0000 complete [10557.920875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10557.948460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10557.957045] Lustre: Skipped 2 previous similar messages [10562.536693] Lustre: server umount lustre-MDT0000 complete [10563.041493] LustreError: 271214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10563.063925] LustreError: 271214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10566.984014] LustreError: 267884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847287 with bad export cookie 12679908560699801849 [10566.989962] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10567.003719] LustreError: 267884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10567.347970] Lustre: server umount lustre-MDT0001 complete [10587.050324] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [10599.745881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10600.455585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10600.465108] Lustre: Skipped 2 previous similar messages [10605.899527] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10615.322580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10620.208866] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10623.518946] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10630.102287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10636.835394] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10647.073213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10647.485185] Lustre: lustre-OST0001: new disk, initializing [10647.497190] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10647.516043] Lustre: 275118:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10648.718099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10648.740426] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10648.953928] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10655.531907] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10666.526507] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10670.529300] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10676.777745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10676.808081] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10684.905913] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10684.918538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10684.924509] Lustre: Skipped 4 previous similar messages [10684.940842] Lustre: Skipped 3 previous similar messages [10690.567362] Lustre: server umount lustre-MDT0000 complete [10694.544577] LustreError: 275116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847415 with bad export cookie 12679908560699802731 [10694.558837] LustreError: 275116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10694.563649] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10695.014116] Lustre: server umount lustre-MDT0001 complete [10709.589103] Lustre: server umount lustre-OST0000 complete [10711.391160] Lustre: 250584:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847416/real 1788847416] req@ffff98030a852a00 x1875741674555904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788847432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10711.427995] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10713.932516] Lustre: server umount lustre-OST0001 complete [10734.897773] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [10746.665676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10747.131857] LustreError: 278174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10747.159360] LustreError: 278174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10747.310604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10747.321819] Lustre: Skipped 3 previous similar messages [10751.914953] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10761.322348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10765.864472] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10768.638437] Lustre: 279316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10774.992768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10781.923678] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10791.307353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10792.713988] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10792.765391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10798.836245] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10808.390756] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10813.098921] Lustre: 281196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10819.757746] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10819.855185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10830.276075] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 02:05:49 (1788847549) [10848.737422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10848.750846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10848.773153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10848.777342] Lustre: Skipped 4 previous similar messages [10852.661758] Lustre: server umount lustre-MDT0000 complete [10858.101697] LustreError: 278154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847579 with bad export cookie 12679908560699809507 [10858.107984] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10858.122357] LustreError: 278154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10864.739994] Lustre: server umount lustre-MDT0001 complete [10875.365659] Lustre: server umount lustre-OST0000 complete [10879.243532] Lustre: server umount lustre-OST0001 complete [10887.255912] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [10902.720871] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [10955.708291] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [10966.392764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10966.652321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10966.682664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10966.800875] Lustre: lustre-MDT0000: new disk, initializing [10966.954235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10972.428915] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10982.715160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10982.809913] Lustre: 285465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10983.134394] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10987.749087] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10999.890568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11000.211034] Lustre: 287099:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11002.031809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11006.685883] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11017.805981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11017.939860] Lustre: 287970:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11018.029987] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11018.037512] Lustre: Skipped 6 previous similar messages [11019.977749] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11019.991230] Lustre: Skipped 1 previous similar message [11020.161894] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11025.091240] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11034.464202] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11038.036180] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11038.045203] Lustre: Skipped 1 previous similar message [11051.785925] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 02:09:32 (1788847772) [11055.080152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11055.095980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11055.106256] Lustre: Skipped 5 previous similar messages [11055.110733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11055.117557] Lustre: Skipped 8 previous similar messages [11059.894861] Lustre: server umount lustre-MDT0000 complete [11061.218127] LustreError: 286267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11061.249052] LustreError: 286267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [11063.485452] LustreError: 287098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847784 with bad export cookie 12679908560699816934 [11063.496984] LustreError: 287098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11063.497542] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11063.842508] Lustre: server umount lustre-MDT0001 complete [11077.957957] Lustre: server umount lustre-OST0000 complete [11091.828306] Lustre: server umount lustre-OST0001 complete [11100.903045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11106.058566] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11116.126247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11122.813900] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11130.690792] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11137.119268] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11144.427445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11151.030305] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11156.838280] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11174.959155] loop1: detected capacity change from 0 to 204800000 [11175.166021] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11175.177417] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11175.186453] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11180.745066] loop1: detected capacity change from 0 to 204800000 [11186.655118] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11186.782052] LustreError: 290205:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs23432: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11186.793956] LustreError: 290205:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs23432 [11186.798081] LustreError: 290205:0:(mgs_handler.c:517:mgs_target_reg()) fs23432-MDT0001: Error checking target index: rc = -13 [11186.808909] LustreError: fs23432-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11186.816901] LustreError: 293932:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11186.823759] LustreError: 293932:0:(tgt_mount.c:2132:server_put_super()) no obd fs23432-MDT0001 [11186.827762] LustreError: 293932:0:(tgt_mount.c:116:server_deregister_mount()) fs23432-MDT0001 not registered [11186.858569] Lustre: server umount fs23432-MDT0001 complete [11186.861727] LustreError: 293932:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11195.734901] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11195.944249] Lustre: Setting parameter fs23432-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs23432-MDT0001 [11196.033556] Lustre: srv-fs23432-MDT0001: No data found on store. Initialize space. [11196.042940] Lustre: Skipped 4 previous similar messages [11196.182499] Lustre: fs23432-MDT0001: new disk, initializing [11196.191465] Lustre: Skipped 3 previous similar messages [11196.312269] Lustre: srv-fs23432-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11196.326129] Lustre: 294399:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23432-MDT0001-mdtlov: can't create new llog: rc = -115 [11196.341356] LustreError: 294399:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23432-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11201.662679] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11208.147947] Lustre: server umount fs23432-MDT0001 complete [11222.712246] loop1: detected capacity change from 0 to 204800000 [11228.658220] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11228.894930] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11229.261083] Lustre: 295969:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23432-MDT0001-mdtlov: can't create new llog: rc = -115 [11229.276030] LustreError: 295969:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23432-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11234.100293] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11240.175890] Lustre: server umount fs23432-MDT0001 complete [11254.256644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11254.268785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11254.284623] Lustre: Skipped 4 previous similar messages [11254.303412] Lustre: Skipped 6 previous similar messages [11256.891879] Lustre: server umount lustre-MDT0000 complete [11268.035346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11268.208719] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11273.531266] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11273.715289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11282.472205] loop1: detected capacity change from 0 to 204800000 [11288.418132] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11289.077985] Lustre: 298419:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23432-MDT0001-mdtlov: can't create new llog: rc = -115 [11289.090607] LustreError: 298419:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23432-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11294.034194] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11298.230568] Lustre: server umount fs23432-MDT0001 complete [11319.791018] LustreError: 292047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11319.801297] LustreError: 292047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 106 previous similar messages [11324.898161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11324.921049] LustreError: Skipped 1 previous similar message [11336.159163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11336.412317] Lustre: server umount lustre-OST0000 complete [11347.690675] Lustre: server umount lustre-MDT0000 complete [11352.680518] LustreError: 290196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848073 with bad export cookie 12679908560699819615 [11352.682400] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11352.696694] LustreError: 290196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11353.027273] Lustre: server umount lustre-MDT0001 complete [11365.713841] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [11368.714663] Key type lgssc unregistered [11369.117204] LNet: 300761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11369.129141] LNetError: 300761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11369.152060] LNet: Removed LNI 192.168.204.116@tcp [11370.366393] Key type .llcrypt unregistered [11370.369712] Key type ._llcrypt unregistered [11393.354516] Key type ._llcrypt registered [11393.357161] Key type .llcrypt registered [11393.546531] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [11395.303921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11395.332417] alg: No test for adler32 (adler32-zlib) [11396.587855] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [11396.969888] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [11398.671573] Key type lgssc registered [11400.366223] Lustre: Echo OBD driver; http://www.lustre.org/ [11409.510854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11409.577716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11411.279536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11416.306123] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11421.667660] LustreError: 302034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11421.693158] LustreError: 302034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11424.540581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11424.853925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11429.181347] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11436.879569] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11442.270238] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11449.482297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11450.127631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11456.308212] LustreError: 303860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11457.285079] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11461.614037] LustreError: 303860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11466.723056] LustreError: 304168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11467.235585] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11474.922894] LustreError: 303862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11474.940038] LustreError: 303862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11485.156187] LustreError: 303862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11485.186357] LustreError: 303862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11489.113444] loop1: detected capacity change from 0 to 204800000 [11489.332156] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11489.344413] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11489.354649] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11495.126180] loop1: detected capacity change from 0 to 204800000 [11500.817756] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11500.994549] Lustre: fs11769-OST0000: new disk, initializing [11501.003420] Lustre: srv-fs11769-OST0000: No data found on store. Initialize space. [11501.009510] Lustre: 305766:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11501.171965] Lustre: fs11769-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11505.651918] LustreError: 304168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11505.685408] LustreError: 304168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11508.679551] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11518.912850] Lustre: server umount fs11769-OST0000 complete [11531.986338] loop1: detected capacity change from 0 to 204800000 [11536.363098] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11537.911463] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11538.199225] Lustre: fs11769-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11541.477827] LustreError: 303861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11541.508631] LustreError: 303861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [11544.379308] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11551.736840] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11553.734347] loop2: detected capacity change from 0 to 204800000 [11553.975933] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11553.992720] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11554.006721] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11559.734541] loop2: detected capacity change from 0 to 204800000 [11565.896706] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11566.038924] LustreError: fs11769-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11566.055118] LustreError: fs11769-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11566.067738] LustreError: 308808:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11566.075328] LustreError: 308808:0:(tgt_mount.c:2132:server_put_super()) no obd fs11769-OST0001 [11566.079684] LustreError: 308808:0:(tgt_mount.c:116:server_deregister_mount()) fs11769-OST0001 not registered [11566.123192] Lustre: server umount fs11769-OST0001 complete [11566.125411] LustreError: 308808:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11579.318554] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11579.682525] Lustre: fs11769-OST0001: new disk, initializing [11579.691428] Lustre: srv-fs11769-OST0001: No data found on store. Initialize space. [11579.706586] Lustre: 309359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11579.821249] Lustre: fs11769-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11586.589679] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11595.562716] Lustre: server umount fs11769-OST0000 complete [11605.240704] Lustre: server umount fs11769-OST0001 complete [11608.047507] LustreError: 303862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11608.081113] LustreError: 303862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11617.925285] loop1: detected capacity change from 0 to 204800000 [11623.624681] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11623.892614] Lustre: fs11769-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11630.092964] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11637.474604] loop2: detected capacity change from 0 to 204800000 [11643.733268] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11650.994303] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11661.200581] Lustre: server umount fs11769-OST0000 complete [11671.270527] Lustre: server umount fs11769-OST0001 complete [11684.842520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11684.851926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11684.867966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11684.869478] Lustre: Skipped 1 previous similar message [11684.890916] Lustre: Skipped 1 previous similar message [11689.963505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11689.968790] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11690.941845] Lustre: server umount lustre-OST0000 complete [11696.102268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11696.127284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11696.143556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11696.148847] Lustre: Skipped 1 previous similar message [11700.204924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11700.205113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11700.216717] Lustre: Skipped 1 previous similar message [11701.521956] Lustre: server umount lustre-MDT0000 complete [11705.874578] LustreError: 302012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848426 with bad export cookie 14051826185129327892 [11705.891765] LustreError: 302012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11705.892906] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11706.694477] Lustre: server umount lustre-MDT0001 complete [11722.111737] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [11725.545943] Key type lgssc unregistered [11726.072071] LNet: 315284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11726.085089] LNetError: 315284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11726.107724] LNet: Removed LNI 192.168.204.116@tcp [11727.314774] Key type .llcrypt unregistered [11727.317779] Key type ._llcrypt unregistered [11767.107443] Key type ._llcrypt registered [11767.112991] Key type .llcrypt registered [11767.253630] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [11787.518154] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [11789.017016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11789.268344] alg: No test for adler32 (adler32-zlib) [11790.687733] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [11791.122624] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [11792.832833] Key type lgssc registered [11794.789711] Lustre: Echo OBD driver; http://www.lustre.org/ [11844.105412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11844.153700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11845.400043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11845.429928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11845.507692] Lustre: lustre-MDT0000: new disk, initializing [11845.593921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11845.609696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11850.048361] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11861.232790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11861.398919] Lustre: 319144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11861.459722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11861.463286] Lustre: Skipped 1 previous similar message [11861.529313] Lustre: lustre-MDT0001: new disk, initializing [11861.643266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11861.678675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11861.685231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11866.525414] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11875.455675] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11880.891666] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11888.247903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11888.497746] Lustre: lustre-OST0000: new disk, initializing [11888.501677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11888.517094] Lustre: 321113:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11888.608667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11894.336484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11894.359943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11894.445320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11895.724970] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11905.028421] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11907.050856] Lustre: Mounted lustre-client - version 2.17.57_113_gf5ca795 [11908.142652] Lustre: Unmounted lustre-client [11917.292317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11917.310451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11917.323800] Lustre: Skipped 1 previous similar message [11922.405803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11922.411744] Lustre: Skipped 1 previous similar message [11923.091819] Lustre: server umount lustre-OST0000 complete [11927.520931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11927.532753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11927.544640] Lustre: Skipped 2 previous similar messages [11927.563276] Lustre: Skipped 1 previous similar message [11932.641947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11932.651520] Lustre: Skipped 1 previous similar message [11937.762139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11937.786801] Lustre: Skipped 1 previous similar message [11941.343652] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11941.588603] Lustre: server umount lustre-MDT0000 complete [11945.860907] LustreError: 319137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848666 with bad export cookie 4375905890340368417 [11945.875032] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11946.410545] Lustre: server umount lustre-MDT0001 complete [11958.093483] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [11960.225386] LNet: 1 local NIs in recovery (showing 1): 192.168.204.116@tcp [11961.014827] Key type lgssc unregistered [11961.342093] LNet: 323275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11961.357515] LNetError: 323275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12039.209059] LNet: Removed LNI 192.168.204.116@tcp [12040.138511] Key type .llcrypt unregistered [12040.141197] Key type ._llcrypt unregistered [12058.590127] Key type ._llcrypt registered [12058.592855] Key type .llcrypt registered [12058.716725] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [12075.470267] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [12076.463582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12076.613309] alg: No test for adler32 (adler32-zlib) [12077.610573] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [12077.926974] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [12079.639134] Key type lgssc registered [12080.701773] Lustre: Echo OBD driver; http://www.lustre.org/ [12113.106816] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:27:13 (1788848833) [12120.750162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12120.774246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12122.106032] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12122.134206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12122.203444] Lustre: lustre-MDT0000: new disk, initializing [12122.318741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12122.354085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12128.247350] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12139.626144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12139.745634] Lustre: 326496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12139.816925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12139.823273] Lustre: Skipped 1 previous similar message [12139.937390] Lustre: lustre-MDT0001: new disk, initializing [12140.034978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12140.070784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12140.082762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12145.376017] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12155.091854] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12161.486029] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12168.491135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12168.724700] Lustre: lustre-OST0000: new disk, initializing [12168.728291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12168.733462] Lustre: 328463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12168.813205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12174.614455] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12174.973354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12174.987900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12175.083372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12185.785956] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12200.930103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12200.931631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12200.957258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12206.053812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12206.065683] Lustre: Skipped 2 previous similar messages [12207.023637] Lustre: server umount lustre-OST0000 complete [12211.172904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12211.174657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12211.188745] Lustre: Skipped 2 previous similar messages [12211.200032] Lustre: Skipped 1 previous similar message [12216.288800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12217.013624] Lustre: server umount lustre-MDT0000 complete [12220.613753] LustreError: 326489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848941 with bad export cookie 10361873528409425733 [12220.621216] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12220.632843] LustreError: 326489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12221.065689] Lustre: server umount lustre-MDT0001 complete [12231.507438] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [12234.098783] Key type lgssc unregistered [12234.555124] LNet: 330369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12234.562891] LNetError: 330369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12234.593066] LNet: Removed LNI 192.168.204.116@tcp [12235.441220] Key type .llcrypt unregistered [12235.443209] Key type ._llcrypt unregistered [12255.337815] Key type ._llcrypt registered [12255.354136] Key type .llcrypt registered [12255.482081] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [12256.540768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12256.557568] alg: No test for adler32 (adler32-zlib) [12257.688329] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [12257.909278] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [12259.591360] Key type lgssc registered [12260.786803] Lustre: Echo OBD driver; http://www.lustre.org/ [12267.831450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12267.869847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12269.400867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12273.950870] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12279.794764] LustreError: 331645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12279.811539] LustreError: 331645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12282.898348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12283.305192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12287.971596] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12295.068397] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12300.669548] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12308.445284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12308.792839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12315.032244] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12322.243661] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12330.742558] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12331.914199] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12335.592019] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12335.600434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12335.603702] Lustre: Skipped 2 previous similar messages [12340.188213] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.16@tcp (stopping) [12340.195461] Lustre: Skipped 2 previous similar messages [12341.765897] Lustre: server umount lustre-MDT0000 complete [12345.263053] LustreError: 333538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12345.416753] LustreError: 331625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849066 with bad export cookie 17508448000078869934 [12345.437932] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12345.440817] LustreError: 331625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12345.826586] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12345.853268] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12345.865300] Lustre: Skipped 3 previous similar messages [12350.383977] LustreError: 331640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12350.413204] LustreError: 331640:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12350.429704] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.16@tcp (stopping) [12351.866677] Lustre: server umount lustre-MDT0001 complete [12355.608982] Lustre: server umount lustre-OST0000 complete [12365.157635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12369.511712] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12372.286740] Lustre: 335825:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.116@tcp [12374.717496] Lustre: server umount MGS complete [12383.286109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12383.700323] LustreError: 336397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12383.722774] LustreError: 336397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12383.776335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12388.026588] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12388.836872] LustreError: 336398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12395.049709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12395.368850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12399.318899] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12407.083923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12407.406475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12412.940951] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12422.059906] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12424.067540] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12430.682315] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12432.563190] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12436.451910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12436.454295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12436.466748] Lustre: Skipped 1 previous similar message [12436.479503] Lustre: Skipped 2 previous similar messages [12442.352163] Lustre: server umount lustre-OST0000 complete [12446.691027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12446.700923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12446.711884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12446.721629] Lustre: Skipped 3 previous similar messages [12452.709425] Lustre: server umount lustre-MDT0000 complete [12456.455031] LustreError: 339081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849177 with bad export cookie 17508448000078870774 [12456.470553] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12456.488254] LustreError: 339081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12456.871972] Lustre: server umount lustre-MDT0001 complete [12467.041859] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [12469.581712] Key type lgssc unregistered [12469.786027] LNet: 339832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12469.795300] LNetError: 339832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12469.814373] LNet: Removed LNI 192.168.204.116@tcp [12470.518181] Key type .llcrypt unregistered [12470.519863] Key type ._llcrypt unregistered [12488.036881] Key type ._llcrypt registered [12488.040925] Key type .llcrypt registered [12488.171905] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [12503.684146] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [12505.090403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12505.129213] alg: No test for adler32 (adler32-zlib) [12506.182978] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [12506.434853] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [12508.095356] Key type lgssc registered [12509.220427] Lustre: Echo OBD driver; http://www.lustre.org/ [12541.304562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12541.357153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12542.704777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12542.742997] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12542.875441] Lustre: lustre-MDT0000: new disk, initializing [12542.971825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12543.011938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12547.421110] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12558.332845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12558.481036] Lustre: 342989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12558.561923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12558.566169] Lustre: Skipped 1 previous similar message [12558.679532] Lustre: lustre-MDT0001: new disk, initializing [12558.769568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12558.810674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12558.826682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12562.886483] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12572.138431] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12576.749899] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12583.652301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12583.928317] Lustre: lustre-OST0000: new disk, initializing [12583.938148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12583.955842] Lustre: 344958:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12584.043843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12590.688843] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12593.219386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12593.228385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12593.332678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12600.199322] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12604.445117] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12604.603604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12609.058662] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12609.320868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12614.119296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12614.128944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12614.148764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12614.154086] Lustre: Skipped 1 previous similar message [12617.399932] Lustre: server umount lustre-OST0000 complete [12624.353118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12624.376395] Lustre: Skipped 1 previous similar message [12624.384123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12624.391793] Lustre: Skipped 1 previous similar message [12627.174202] Lustre: server umount lustre-MDT0000 complete [12630.345730] LustreError: 342982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849351 with bad export cookie 6367105170548807233 [12630.348691] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12630.356476] LustreError: 342982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12630.632844] Lustre: server umount lustre-MDT0001 complete [12638.291280] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:35:58 (1788849358) [12652.241189] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [12660.615233] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [12689.341473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12689.511562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12689.533696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12689.594061] Lustre: lustre-MDT0000: new disk, initializing [12689.643748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12689.659042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12692.978248] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12701.801344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12701.938219] Lustre: 349712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12702.003134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12702.006844] Lustre: Skipped 1 previous similar message [12702.085741] Lustre: lustre-MDT0001: new disk, initializing [12702.180312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12702.222473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12702.237981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12706.018223] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12715.191860] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12720.251886] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12726.547940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12726.826745] Lustre: lustre-OST0000: new disk, initializing [12726.833020] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12726.839963] Lustre: 351684:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12726.957931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12732.168844] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12737.091850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12737.109463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12737.237819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12741.070657] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12771.547386] Lustre: server umount lustre-OST0000 complete [12778.465947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12778.476294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12778.484462] Lustre: Skipped 2 previous similar messages [12778.495429] Lustre: Skipped 1 previous similar message [12783.587592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12783.594231] Lustre: Skipped 1 previous similar message [12784.683816] Lustre: server umount lustre-MDT0000 complete [12787.552120] LustreError: 349707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849508 with bad export cookie 6367105170548808108 [12787.561819] LustreError: 349707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12787.563737] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12798.032489] Lustre: server umount lustre-MDT0001 complete [12806.646890] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [12808.806891] Key type lgssc unregistered [12809.035751] LNet: 353835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12809.048596] LNetError: 353835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12809.062528] LNet: Removed LNI 192.168.204.116@tcp [12809.795670] Key type .llcrypt unregistered [12809.797377] Key type ._llcrypt unregistered [12877.781317] Key type ._llcrypt registered [12877.783933] Key type .llcrypt registered [12877.906311] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [12878.172380] Key type .llcrypt unregistered [12878.176849] Key type ._llcrypt unregistered [12935.748234] Key type ._llcrypt registered [12935.750751] Key type .llcrypt registered [12935.839477] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [12936.065366] Key type .llcrypt unregistered [12936.067723] Key type ._llcrypt unregistered [12958.630184] Key type ._llcrypt registered [12958.631893] Key type .llcrypt registered [12958.737450] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [12958.993840] Key type .llcrypt unregistered [12958.998089] Key type ._llcrypt unregistered [12981.833720] Key type ._llcrypt registered [12981.835928] Key type .llcrypt registered [12981.926900] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [12982.248501] Key type .llcrypt unregistered [12982.253130] Key type ._llcrypt unregistered [13038.097091] Key type ._llcrypt registered [13038.098997] Key type .llcrypt registered [13038.192744] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [13038.434333] Key type .llcrypt unregistered [13038.437799] Key type ._llcrypt unregistered [13060.995858] Key type ._llcrypt registered [13060.998188] Key type .llcrypt registered [13061.072586] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [13061.295416] Key type .llcrypt unregistered [13061.300697] Key type ._llcrypt unregistered [13083.371651] Key type ._llcrypt registered [13083.374023] Key type .llcrypt registered [13083.466544] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [13083.727869] Key type .llcrypt unregistered [13083.733946] Key type ._llcrypt unregistered [13105.311912] Key type ._llcrypt registered [13105.315038] Key type .llcrypt registered [13105.408522] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [13105.618274] Key type .llcrypt unregistered [13105.622307] Key type ._llcrypt unregistered [13126.571410] Key type ._llcrypt registered [13126.573735] Key type .llcrypt registered [13126.658450] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [13126.917856] Key type .llcrypt unregistered [13126.920199] Key type ._llcrypt unregistered [13151.542542] Key type ._llcrypt registered [13151.547651] Key type .llcrypt registered [13151.653908] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [13151.903541] Key type .llcrypt unregistered [13151.905687] Key type ._llcrypt unregistered [13200.418465] Key type ._llcrypt registered [13200.421634] Key type .llcrypt registered [13200.535359] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [13200.796318] Key type .llcrypt unregistered [13200.800228] Key type ._llcrypt unregistered [13222.702540] Key type ._llcrypt registered [13222.704682] Key type .llcrypt registered [13222.783406] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [13223.012332] Key type .llcrypt unregistered [13223.016154] Key type ._llcrypt unregistered [13243.633616] Key type ._llcrypt registered [13243.635602] Key type .llcrypt registered [13243.709452] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [13243.910546] Key type .llcrypt unregistered [13243.913468] Key type ._llcrypt unregistered [13280.811834] Key type ._llcrypt registered [13280.815121] Key type .llcrypt registered [13280.882696] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [13291.901519] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [13292.494864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13292.598198] alg: No test for adler32 (adler32-zlib) [13293.583293] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [13293.766357] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [13295.399493] Key type lgssc registered [13296.238295] Lustre: Echo OBD driver; http://www.lustre.org/ [13322.034591] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13322.064878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13323.247578] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13323.278766] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13323.353141] Lustre: tst250fs-MDT0000: new disk, initializing [13323.422634] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13323.440066] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13326.775155] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13334.565143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13334.656453] Lustre: 373722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13334.712319] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13334.719696] Lustre: Skipped 1 previous similar message [13334.791849] Lustre: tst250fs-MDT0001: new disk, initializing [13334.869574] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13334.898190] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13334.905340] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13338.417165] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13342.334324] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13348.022914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13348.252723] Lustre: tst250fs-OST0000: new disk, initializing [13348.255777] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13348.261405] Lustre: 375352:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13348.335510] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13350.567254] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13350.577062] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13350.677380] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13353.157986] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13360.886317] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13361.012707] Lustre: tst250fs-OST0001: new disk, initializing [13361.018525] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13361.024539] Lustre: 376225:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13361.086260] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13365.313441] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13366.837948] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13366.844388] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13366.899222] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13369.569175] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13386.211122] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13386.216119] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13386.228259] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13387.233297] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13387.233608] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13387.246472] Lustre: Skipped 2 previous similar messages [13387.257349] Lustre: Skipped 3 previous similar messages [13391.103706] Lustre: server umount tst250fs-MDT0000 complete [13393.545584] LustreError: 374517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850114 with bad export cookie 16946542306503541064 [13393.547561] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13393.553798] LustreError: 374517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13393.803988] Lustre: server umount tst250fs-MDT0001 complete [13405.723619] Lustre: server umount tst250fs-OST0000 complete [13418.524460] Lustre: server umount tst250fs-OST0001 complete [13430.624671] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [13435.628234] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [13456.210536] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 02:49:36 (1788850176) [13457.339518] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13467.579233] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [13473.248963] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [13492.010374] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13179 sec ========= 02:50:12 (1788850212) [13493.049904] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:50:13 (1788850213) === [13495.420852] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:50:16 (1788850216) === [13518.226874] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [13519.801323] Key type lgssc unregistered [13519.982518] LNet: 385354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13519.990932] LNetError: 385354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13520.005767] LNet: Removed LNI 192.168.204.116@tcp [13520.487154] Key type .llcrypt unregistered [13520.489710] Key type ._llcrypt unregistered