[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 560820290 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002388] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007415] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011063] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.013068] *** VALIDATE selinux *** [ 0.022057] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026376] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029091] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030124] *** VALIDATE tmpfs *** [ 0.032121] *** VALIDATE proc *** [ 0.033004] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035227] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036116] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038021] Spectre V2 : User space: Vulnerable [ 0.039005] Speculative Store Bypass: Vulnerable [ 0.042226] debug: unmapping init [mem 0xffffffff9fc59000-0xffffffff9fc60fff] [ 0.045197] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046748] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.047909] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053352] rcu: Hierarchical SRCU implementation. [ 0.055281] smp: Bringing up secondary CPUs ... [ 0.056444] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.060404] smp: Brought up 1 node, 4 CPUs [ 0.062017] smpboot: Max logical packages: 1 [ 0.063024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.233614] node 0 deferred pages initialised in 168ms [ 0.235211] devtmpfs: initialized [ 0.236323] x86/mm: Memory block size: 128MB [ 0.240214] gcov: version magic: 0x41383552 [ 0.242587] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.245086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.247367] pinctrl core: initialized pinctrl subsystem [ 0.248206] [ 0.248697] ************************************************************* [ 0.251016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253010] ** ** [ 0.254009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.255010] ** ** [ 0.257011] ** This means that this kernel is built to expose internal ** [ 0.258011] ** IOMMU data structures, which may compromise security on ** [ 0.259010] ** your system. ** [ 0.260009] ** ** [ 0.262012] ** If you see this message and you are not debugging the ** [ 0.264013] ** kernel, report this immediately to your vendor! ** [ 0.266019] ** ** [ 0.268017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.270018] ************************************************************* [ 0.272804] NET: Registered protocol family 16 [ 0.274580] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.277095] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.278047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.280425] cpuidle: using governor menu [ 0.281706] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.282539] PCI: Using configuration type 1 for base access [ 0.283104] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.291288] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.293027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.296248] cryptd: max_cpu_qlen set to 1000 [ 0.300194] ACPI: Added _OSI(Module Device) [ 0.301016] ACPI: Added _OSI(Processor Device) [ 0.303016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.304014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.308073] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.312418] ACPI: Interpreter enabled [ 0.313050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.315024] ACPI: Using IOAPIC for interrupt routing [ 0.316247] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.320356] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.327892] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.329047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.331019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.333147] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.337272] acpiphp: Slot [2] registered [ 0.338155] acpiphp: Slot [5] registered [ 0.339120] acpiphp: Slot [6] registered [ 0.340096] acpiphp: Slot [7] registered [ 0.341098] acpiphp: Slot [8] registered [ 0.343184] acpiphp: Slot [9] registered [ 0.344094] acpiphp: Slot [10] registered [ 0.346172] acpiphp: Slot [3] registered [ 0.347145] acpiphp: Slot [4] registered [ 0.348037] acpiphp: Slot [11] registered [ 0.348960] acpiphp: Slot [12] registered [ 0.350069] acpiphp: Slot [13] registered [ 0.350891] acpiphp: Slot [14] registered [ 0.351051] acpiphp: Slot [15] registered [ 0.351947] acpiphp: Slot [16] registered [ 0.353064] acpiphp: Slot [17] registered [ 0.353874] acpiphp: Slot [18] registered [ 0.355093] acpiphp: Slot [19] registered [ 0.355868] acpiphp: Slot [20] registered [ 0.356076] acpiphp: Slot [21] registered [ 0.356892] acpiphp: Slot [22] registered [ 0.358087] acpiphp: Slot [23] registered [ 0.359021] acpiphp: Slot [24] registered [ 0.359898] acpiphp: Slot [25] registered [ 0.361056] acpiphp: Slot [26] registered [ 0.361958] acpiphp: Slot [27] registered [ 0.362075] acpiphp: Slot [28] registered [ 0.363000] acpiphp: Slot [29] registered [ 0.363876] acpiphp: Slot [30] registered [ 0.365066] acpiphp: Slot [31] registered [ 0.365829] PCI host bridge to bus 0000:00 [ 0.367020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.368020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.370022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.371018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.372019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.374029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.375212] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.378334] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.381315] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.389020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.393177] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.396026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.398015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.400035] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.402662] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.405657] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.407046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.408646] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.411844] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.420995] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.426021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.430158] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.436025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.442034] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.455030] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.464774] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.476020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.487025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.503022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.512107] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.517022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.522024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.535020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.544000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.550020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.555019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.568028] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.577763] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.583028] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.594023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.605029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.612962] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.617018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.621016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.631014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.640423] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.641258] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.643352] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.645428] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.647194] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.652060] iommu: Default domain type: Passthrough [ 0.653497] SCSI subsystem initialized [ 0.654153] ACPI: bus type USB registered [ 0.655089] usbcore: registered new interface driver usbfs [ 0.656061] usbcore: registered new interface driver hub [ 0.657095] usbcore: registered new device driver usb [ 0.658164] pps_core: LinuxPPS API ver. 1 registered [ 0.660010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.661092] PTP clock support registered [ 0.663159] EDAC MC: Ver: 3.0.0 [ 0.664339] PCI: Using ACPI for IRQ routing [ 0.666774] NetLabel: Initializing [ 0.667009] NetLabel: domain hash size = 128 [ 0.668030] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.669164] NetLabel: unlabeled traffic allowed by default [ 0.671128] vgaarb: loaded [ 0.672232] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.673011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.677490] clocksource: Switched to clocksource kvm-clock [ 0.772882] VFS: Disk quotas dquot_6.6.0 [ 0.774357] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.776605] *** VALIDATE ramfs *** [ 0.777477] *** VALIDATE hugetlbfs *** [ 0.779437] pnp: PnP ACPI init [ 0.781376] pnp: PnP ACPI: found 6 devices [ 0.795338] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.797458] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.799064] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.800366] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.802399] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.804641] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.806780] NET: Registered protocol family 2 [ 0.808547] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.812825] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.815310] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.819114] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.821328] TCP: Hash tables configured (established 65536 bind 65536) [ 0.823159] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.824931] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.826514] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.828409] NET: Registered protocol family 1 [ 0.830957] RPC: Registered named UNIX socket transport module. [ 0.832806] RPC: Registered udp transport module. [ 0.834287] RPC: Registered tcp transport module. [ 0.835732] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.837733] NET: Registered protocol family 44 [ 0.839189] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.841121] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.842774] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.844792] PCI: CLS 0 bytes, default 64 [ 0.846239] Unpacking initramfs... [ 2.295966] debug: unmapping init [mem 0xffff8911bcc54000-0xffff8911bffbffff] [ 2.302035] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.304291] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.308501] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.798657] Initialise system trusted keyrings [ 2.800392] Key type blacklist registered [ 2.802048] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.809084] zbud: loaded [ 2.811953] *** VALIDATE nfs *** [ 2.813065] *** VALIDATE nfs4 *** [ 2.814260] pstore: using deflate compression [ 2.817046] Platform Keyring initialized [ 2.920426] NET: Registered protocol family 38 [ 2.922083] Key type asymmetric registered [ 2.923317] Asymmetric key parser 'x509' registered [ 2.924922] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.927529] io scheduler mq-deadline registered [ 2.928908] io scheduler kyber registered [ 2.930265] io scheduler bfq registered [ 2.932064] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.934924] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.937211] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.939427] ACPI: Power Button [PWRF] [ 2.944909] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.951928] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.965516] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.972230] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.986637] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.013043] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.040261] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.043843] Non-volatile memory driver v1.3 [ 3.045347] Linux agpgart interface v0.103 [ 3.075470] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.077503] vda: detected capacity change from 0 to 74706944 [ 3.089094] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.090906] vdb: detected capacity change from 0 to 1073741824 [ 3.101331] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.103209] vdc: detected capacity change from 0 to 2621440000 [ 3.114982] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.116994] vdd: detected capacity change from 0 to 2621440000 [ 3.127936] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.129910] vde: detected capacity change from 0 to 4294967296 [ 3.141782] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.143843] vdf: detected capacity change from 0 to 4294967296 [ 3.148755] libphy: Fixed MDIO Bus: probed [ 3.153184] usbcore: registered new interface driver usbserial_generic [ 3.155099] usbserial: USB Serial support registered for generic [ 3.157823] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.162210] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.164301] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.166963] mousedev: PS/2 mouse device common for all mice [ 3.169449] rtc_cmos 00:05: RTC can wake from S4 [ 3.170950] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.171856] rtc_cmos 00:05: registered as rtc0 [ 3.176169] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.177831] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.178608] intel_pstate: CPU model not supported [ 3.182191] hid: raw HID events driver (C) Jiri Kosina [ 3.184175] usbcore: registered new interface driver usbhid [ 3.185047] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.185520] usbhid: USB HID core driver [ 3.189036] drop_monitor: Initializing network drop monitor service [ 3.190438] Initializing XFRM netlink socket [ 3.191852] NET: Registered protocol family 10 [ 3.194279] Segment Routing with IPv6 [ 3.195644] NET: Registered protocol family 17 [ 3.197668] mpls_gso: MPLS GSO support [ 3.205230] RAS: Correctable Errors collector initialized. [ 3.206972] AVX version of gcm_enc/dec engaged. [ 3.208269] AES CTR mode by8 optimization enabled [ 3.274655] sched_clock: Marking stable (3274637948, 0)->(4138998413, -864360465) [ 3.276781] registered taskstats version 1 [ 3.278045] Loading compiled-in X.509 certificates [ 3.278988] zswap: loaded using pool lzo/zbud [ 3.300107] Key type big_key registered [ 3.311292] Key type encrypted registered [ 3.312416] ima: No TPM chip found, activating TPM-bypass! [ 3.313644] ima: Allocated hash algorithm: sha1 [ 3.315181] ima: No architecture policies found [ 3.316296] evm: Initialising EVM extended attributes: [ 3.317457] evm: security.selinux [ 3.318183] evm: security.ima [ 3.318693] evm: security.capability [ 3.319343] evm: HMAC attrs: 0x1 [ 3.320903] rtc_cmos 00:05: setting system clock to 2026-08-11 10:25:31 UTC (1786443931) [ 3.324551] debug: unmapping init [mem 0xffffffffa0c03000-0xffffffffa0dfffff] [ 3.326374] debug: unmapping init [mem 0xffffffff9f982000-0xffffffff9fc58fff] [ 3.334144] Write protecting the kernel read-only data: 28672k [ 3.337082] debug: unmapping init [mem 0xffffffff9e003000-0xffffffff9e1fffff] [ 3.338558] debug: unmapping init [mem 0xffffffff9e914000-0xffffffff9e9fffff] [ 3.368721] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.376239] systemd[1]: Detected virtualization kvm. [ 3.377417] systemd[1]: Detected architecture x86-64. [ 3.379249] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.401018] systemd[1]: No hostname configured. [ 3.402438] systemd[1]: Set hostname to . [ 3.404457] random: systemd: uninitialized urandom read (16 bytes read) [ 3.406920] systemd[1]: Initializing machine ID from random generator. [ 3.445098] random: ln: uninitialized urandom read (6 bytes read) [ 3.525599] random: systemd: uninitialized urandom read (16 bytes read) [ 3.527163] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.532153] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.534270] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.120063] device-mapper: uevent: version 1.0.3 [ 4.121784] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [[ 4.780832] random: fast init done  OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.845383] virtio_net virtio0 ens2: renamed from eth0 [ 4.882521] scsi host0: ata_piix [ 4.897161] scsi host1: ata_piix [ 4.898550] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.900952] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.652938] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.823704] random: crng init done [ 9.825026] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.215726] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.380960] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.596848] SELinux: Disabled at runtime. [ 11.654024] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.663515] systemd[1]: Detected virtualization kvm. [ 11.665143] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.074751] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.076853] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.080464] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.082438] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.084098] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.092836] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.101501] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 12.161301] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.554658] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.887918] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.926888] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.067304] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.093783] EDAC sbridge: Ver: 1.1.2 [ 14.590621] Key type dns_resolver registered [ 14.891612] NFS: Registering the id_resolver key type [ 14.893792] Key type id_resolver registered [ 14.894994] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg138-server login: [ 43.339626] libcfs: loading out-of-tree module taints kernel. [ 43.383845] Key type ._llcrypt registered [ 43.386731] Key type .llcrypt registered [ 43.461732] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 57.745639] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 59.322544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 59.350176] alg: No test for adler32 (adler32-zlib) [ 60.813573] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 61.821358] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 63.600359] Key type lgssc registered [ 65.340667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.249583] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 116.332125] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 128.030029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 128.085888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 129.323154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 129.392682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 129.486649] Lustre: lustre-MDT0000: new disk, initializing [ 129.554971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 129.575614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 133.097948] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 142.905394] hrtimer: interrupt took 4274337 ns [ 145.251224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 145.310567] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 145.331816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 145.333986] Lustre: Skipped 1 previous similar message [ 145.389778] Lustre: lustre-MDT0001: new disk, initializing [ 145.433789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 145.455734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 145.461682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 149.875530] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.211507] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 171.337224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 171.933565] Lustre: lustre-OST0000: new disk, initializing [ 171.938541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 171.945476] Lustre: 8457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 172.047641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.913217] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.382926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 181.391800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 181.475976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 201.764881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.974711] Lustre: lustre-OST0001: new disk, initializing [ 201.978912] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 201.985745] Lustre: 9530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.045508] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.361942] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.486054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 211.503155] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 211.631988] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 225.849368] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.236570] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 243.010431] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing check_logdir /tmp/testlogs/ [ 250.191034] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing yml_node [ 255.279744] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 257.954884] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 260.780339] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 263.229963] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 11 06:29:49 EDT 2026 [ 283.981600] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 285.867806] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 303.590559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.606243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.609280] Lustre: Skipped 2 previous similar messages [ 303.642768] Lustre: Skipped 3 previous similar messages [ 308.705413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.394556] Lustre: server umount lustre-MDT0000 complete [ 313.793126] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786444241 with bad export cookie 4500889699503802962 [ 313.799903] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.801792] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 313.834466] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.843539] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.844844] Lustre: Skipped 2 previous similar messages [ 313.862752] Lustre: Skipped 4 previous similar messages [ 318.946402] LustreError: 6525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.953639] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 318.972428] LustreError: 6525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 319.006616] Lustre: Skipped 1 previous similar message [ 324.064852] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.066493] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.086995] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 324.093587] Lustre: Skipped 1 previous similar message [ 328.160696] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 328.960236] Lustre: server umount lustre-MDT0001 complete [ 332.606261] Lustre: server umount lustre-OST0000 complete [ 336.985177] Lustre: server umount lustre-OST0001 complete [ 344.728874] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 355.777643] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 394.603523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.967681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 395.010932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 395.136527] Lustre: lustre-MDT0000: new disk, initializing [ 395.279681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.306371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 400.880924] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 413.231822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.400374] Lustre: 15089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 413.468934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 413.473684] Lustre: Skipped 1 previous similar message [ 413.573344] Lustre: lustre-MDT0001: new disk, initializing [ 413.683231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 413.734650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 413.745298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 418.764788] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 429.733440] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 435.738421] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 443.423697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 443.739698] Lustre: lustre-OST0000: new disk, initializing [ 443.744464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 443.758714] Lustre: 17058:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 443.872609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.546549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 445.560707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 445.699479] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 452.299890] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 464.017401] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 470.447704] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 470.717836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 476.981985] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 477.213943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 481.761026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 481.769060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.777563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 481.788388] Lustre: Skipped 1 previous similar message [ 485.756528] Lustre: server umount lustre-OST0000 complete [ 492.018363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.039171] Lustre: Skipped 1 previous similar message [ 502.245099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 502.260220] Lustre: Skipped 6 previous similar messages [ 505.314748] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 505.721518] Lustre: server umount lustre-MDT0000 complete [ 507.377081] LustreError: 15095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.425805] LustreError: 15095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 510.160050] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786444438 with bad export cookie 4500889699503804495 [ 510.166530] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 510.174715] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 510.587590] Lustre: server umount lustre-MDT0001 complete [ 517.667733] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 06:34:04 (1786444444) [ 540.435433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.054809] LustreError: 19934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.204471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 546.281237] LustreError: 19935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.813955] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 556.528134] LustreError: 19934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.583351] LustreError: 19934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 556.706781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.570465] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 569.838465] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 575.064637] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 577.226795] Lustre: Failing over lustre-MDT0000 [ 577.504989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 577.509940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.520075] LustreError: 19935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.559427] Lustre: server umount lustre-MDT0000 complete [ 594.403172] Lustre: 3653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786444506/real 1786444506] req@ffff89123d16f100 x1873222292380416/t0(0) o400->MGC192.168.201.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786444522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 594.444488] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.840507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.010471] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 609.192823] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 610.273745] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 610.279442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 610.299251] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 617.490513] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 625.634744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.635853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 625.671578] Lustre: Skipped 1 previous similar message [ 625.672527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 636.384121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 637.108868] Lustre: server umount lustre-MDT0000 complete [ 637.943950] LustreError: 19935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.955652] LustreError: 19935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 643.594552] LustreError: 19915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786444571 with bad export cookie 4500889699503805909 [ 643.609740] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 643.617241] LustreError: 19915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 644.204851] Lustre: server umount lustre-MDT0001 complete [ 659.126905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 660.527850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 660.540807] Lustre: Skipped 2 previous similar messages [ 672.846587] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 689.826573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.660746] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 708.749753] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 716.417828] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 719.891961] Lustre: Failing over lustre-MDT0000 [ 720.248302] Lustre: server umount lustre-MDT0000 complete [ 721.896236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 721.898983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 721.899490] LustreError: 23270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 721.899499] LustreError: 23270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 721.904256] Lustre: Skipped 2 previous similar messages [ 738.165258] Lustre: 3655:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786444650/real 1786444650] req@ffff891102165880 x1873222292413184/t0(0) o400->MGC192.168.201.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786444666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 738.216639] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 740.658305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 748.514258] LustreError: 3652:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff891103369f80 x1873222292415872/t0(0) o250->MGC192.168.201.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 749.085237] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 754.165618] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 754.178742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 754.195871] Lustre: Skipped 1 previous similar message [ 754.214405] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 754.757303] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 765.654877] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 769.827480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.855549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 769.860704] Lustre: Skipped 5 previous similar messages [ 774.627863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 784.352331] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 784.730162] Lustre: server umount lustre-MDT0000 complete [ 788.858293] LustreError: 23256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786444717 with bad export cookie 4500889699503806847 [ 788.869853] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 788.870505] LustreError: 23256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 789.114369] Lustre: server umount lustre-MDT0001 complete [ 799.018266] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 06:38:46 (1786444726) [ 819.043167] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 830.248463] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 878.986571] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 891.426311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 891.659554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 891.688257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 891.793307] Lustre: lustre-MDT0000: new disk, initializing [ 891.864817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 891.873435] Lustre: Skipped 2 previous similar messages [ 891.887734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 896.989953] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 909.686496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.802595] Lustre: 29654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 910.089302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 915.949575] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 927.684766] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 934.762469] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 944.752488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 945.276572] Lustre: 31627:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 946.760208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 946.851644] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 946.944883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 946.969927] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 953.416724] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 963.506367] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 969.257996] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 993.249622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 993.261974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.273095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 993.297204] Lustre: Skipped 2 previous similar messages [ 993.324610] Lustre: Skipped 6 previous similar messages [ 997.208784] Lustre: server umount lustre-OST0000 complete [ 1016.289747] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1016.560069] Lustre: server umount lustre-MDT0000 complete [ 1018.849658] LustreError: 29666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.867329] LustreError: 29666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 1020.527312] LustreError: 31637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786444948 with bad export cookie 4500889699503807253 [ 1020.533945] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1020.537658] LustreError: 31637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1020.979253] Lustre: server umount lustre-MDT0001 complete [ 1032.406851] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1034.923472] Key type lgssc unregistered [ 1035.235070] LNet: 33609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.242490] LNetError: 33609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.262704] LNet: Removed LNI 192.168.201.138@tcp [ 1036.065364] Key type .llcrypt unregistered [ 1036.068072] Key type ._llcrypt unregistered [ 1057.930930] Key type ._llcrypt registered [ 1057.938851] Key type .llcrypt registered [ 1058.036894] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1059.471752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1059.561837] alg: No test for adler32 (adler32-zlib) [ 1060.619213] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 1060.885772] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1062.552543] Key type lgssc registered [ 1063.670745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1071.091926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1071.118086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.533608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.747789] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1081.842101] LustreError: 35031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.860299] LustreError: 35031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1084.756049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.054030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1089.411991] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1096.941670] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1102.459261] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1109.814995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1109.985822] LustreError: 36849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1110.283035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.647714] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1116.733660] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1117.723731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1123.816446] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1136.729738] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 06:44:24 (1786445064) [ 1138.312109] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1139.949603] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1146.030928] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 06:44:32 (1786445072) [ 1154.937674] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 06:44:42 (1786445082) [ 1156.149275] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1163.358492] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1163.367025] Lustre: Skipped 2 previous similar messages [ 1164.911675] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 06:44:52 (1786445092) [ 1167.518721] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1167.524961] Lustre: Skipped 3 previous similar messages [ 1182.383710] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1182.390147] Lustre: Skipped 6 previous similar messages [ 1185.237957] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 06:45:12 (1786445112) [ 1187.865252] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1191.018112] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1198.212455] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 06:45:25 (1786445125) [ 1200.161952] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1200.175529] Lustre: Skipped 3 previous similar messages [ 1212.147663] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 06:45:39 (1786445139) [ 1228.259700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1228.262728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.277700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1230.306322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.310975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1230.311486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1230.338882] Lustre: Skipped 1 previous similar message [ 1231.657666] Lustre: server umount lustre-OST0000 complete [ 1235.431781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.444153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1235.453969] Lustre: Skipped 1 previous similar message [ 1235.465532] Lustre: Skipped 1 previous similar message [ 1240.550388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1240.557479] Lustre: Skipped 1 previous similar message [ 1245.665921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1245.680275] Lustre: Skipped 1 previous similar message [ 1249.248117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1249.401390] Lustre: server umount lustre-MDT0000 complete [ 1250.794916] LustreError: 35025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.817479] LustreError: 35025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1252.628710] LustreError: 38942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786445180 with bad export cookie 8276277767771238516 [ 1252.630971] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1252.637241] LustreError: 38942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1252.912511] Lustre: server umount lustre-MDT0001 complete [ 1261.698141] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1264.526369] Key type lgssc unregistered [ 1264.783833] LNet: 42873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.799055] LNetError: 42873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.836144] LNet: Removed LNI 192.168.201.138@tcp [ 1265.518921] Key type .llcrypt unregistered [ 1265.520404] Key type ._llcrypt unregistered [ 1280.814818] Key type ._llcrypt registered [ 1280.816977] Key type .llcrypt registered [ 1280.891593] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 1293.125560] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1294.104658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1294.121673] alg: No test for adler32 (adler32-zlib) [ 1295.140980] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 1295.340239] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1296.992557] Key type lgssc registered [ 1297.957793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1326.279471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1326.308441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1327.515146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1327.535171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1327.611530] Lustre: lustre-MDT0000: new disk, initializing [ 1327.675534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1327.688912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1330.662683] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1338.644943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.770558] Lustre: 46003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1338.838595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1338.840944] Lustre: Skipped 1 previous similar message [ 1338.891152] Lustre: lustre-MDT0001: new disk, initializing [ 1338.978561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1338.997205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1339.009537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1342.548796] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1349.777534] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1353.954110] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1358.932472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1359.127542] Lustre: lustre-OST0000: new disk, initializing [ 1359.129761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1359.132896] Lustre: 47969:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1359.198599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1364.037307] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1365.024268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1365.032766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1365.123598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1371.380753] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1374.981452] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1375.116140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1378.812975] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1378.995785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1380.837487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1380.842779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.867868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1385.442786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1385.446309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1385.454393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1386.690480] Lustre: server umount lustre-OST0000 complete [ 1390.052318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1390.057410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.071985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.079208] Lustre: Skipped 1 previous similar message [ 1395.680785] Lustre: server umount lustre-MDT0000 complete [ 1398.652847] LustreError: 45993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786445326 with bad export cookie 15999089317978797913 [ 1398.658064] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1398.663119] LustreError: 45993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1398.936108] Lustre: server umount lustre-MDT0001 complete [ 1406.064930] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 06:48:53 (1786445333) [ 1410.579606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.893400] LustreError: 50429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1410.911689] LustreError: 50429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1410.953529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.224979] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1416.162309] LustreError: 50430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.845567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.124173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1424.481307] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1430.537883] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1434.791754] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1440.304509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1440.529905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1444.795828] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1450.067422] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1464.141639] LustreError: 52977:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1464.857414] Lustre: Modifying parameter general.timeout=1 in log params [ 1466.195457] Lustre: Modifying parameter general.timeout=3 in log params [ 1466.198168] Lustre: Skipped 1 previous similar message [ 1468.588428] Lustre: Modifying parameter general.timeout=7 in log params [ 1468.590659] Lustre: Skipped 3 previous similar messages [ 1472.652386] Lustre: Modifying parameter general.timeout=13 in log params [ 1472.655552] Lustre: Skipped 5 previous similar messages [ 1480.887332] Lustre: Modifying parameter general.timeout=28 in log params [ 1480.889398] Lustre: Skipped 14 previous similar messages [ 1497.226811] Lustre: Modifying parameter general.timeout=58 in log params [ 1497.228594] Lustre: Skipped 29 previous similar messages [ 1529.279959] Lustre: Modifying parameter general.timeout=115 in log params [ 1529.283280] Lustre: Skipped 56 previous similar messages [ 1546.910287] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 06:51:14 (1786445474) [ 1549.065818] Lustre: 59893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1550.382133] Lustre: 59991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1552.890271] Lustre: 60202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1552.904981] Lustre: 60202:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1575.165641] Lustre: 61750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1575.171172] Lustre: 61750:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1575.189763] LustreError: 61750:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1578.279972] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 06:51:45 (1786445505) [ 1588.194662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1588.196809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1588.203151] Lustre: Skipped 3 previous similar messages [ 1588.208350] Lustre: Skipped 3 previous similar messages [ 1593.316572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1593.321265] Lustre: Skipped 3 previous similar messages [ 1593.538632] Lustre: server umount lustre-MDT0000 complete [ 1596.084578] LustreError: 50409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786445524 with bad export cookie 15999089317978798802 [ 1596.089598] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1596.096529] LustreError: 50409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1596.318776] Lustre: server umount lustre-MDT0001 complete [ 1609.653659] Lustre: server umount lustre-OST0000 complete [ 1628.766670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1629.033967] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1629.061310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1629.067221] Lustre: Skipped 37 previous similar messages [ 1629.219167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1632.164224] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1638.358951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.421155] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1638.438694] Lustre: 64280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1641.733644] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1647.111637] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1650.838455] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1656.495112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1656.631523] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1660.957601] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1665.798585] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1666.949350] Lustre: 66848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1674.723713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.728091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1674.728529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1674.735122] Lustre: Skipped 1 previous similar message [ 1674.746556] LustreError: Skipped 1 previous similar message [ 1676.389163] Lustre: server umount lustre-OST0000 complete [ 1679.842323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1685.417439] Lustre: server umount lustre-MDT0000 complete [ 1688.107223] LustreError: 65030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786445616 with bad export cookie 15999089317978801511 [ 1688.118776] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1688.122849] LustreError: 65030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1688.353572] Lustre: server umount lustre-MDT0001 complete [ 1695.076196] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1697.112514] Key type lgssc unregistered [ 1697.381574] LNet: 68058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.391468] LNetError: 68058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1697.401684] LNet: Removed LNI 192.168.201.138@tcp [ 1698.107650] Key type .llcrypt unregistered [ 1698.109702] Key type ._llcrypt unregistered [ 1715.216523] Key type ._llcrypt registered [ 1715.218559] Key type .llcrypt registered [ 1715.285326] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1715.880942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1715.930781] alg: No test for adler32 (adler32-zlib) [ 1716.827896] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 1717.007286] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1718.656181] Key type lgssc registered [ 1719.352363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1724.377110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1724.387705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.683837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.983781] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1735.149737] LustreError: 69451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1735.164570] LustreError: 69451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1735.291951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1735.694467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1738.841317] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1744.270980] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1748.354875] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1753.452172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1753.911947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1758.704893] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1764.338368] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1775.704329] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1781.729811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1781.734845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.742963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.827361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.827546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.834882] Lustre: Skipped 1 previous similar message [ 1785.842472] Lustre: Skipped 1 previous similar message [ 1787.872968] Lustre: server umount lustre-MDT0000 complete [ 1790.593911] LustreError: 71303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786445718 with bad export cookie 14611739031782203486 [ 1790.599398] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1790.602495] LustreError: 71303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1790.778496] Lustre: server umount lustre-MDT0001 complete [ 1803.601735] Lustre: server umount lustre-OST0000 complete [ 1821.585333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1821.734361] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1821.759369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1821.925312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.849573] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1830.602538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.654547] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1830.677488] Lustre: 74320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1830.958620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1833.936051] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1839.475253] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1844.151854] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1849.186679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1849.304950] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1849.574586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.738584] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1858.704495] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1859.819830] Lustre: 76891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1860.884193] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1867.749174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.750512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1867.757483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.762118] Lustre: Skipped 1 previous similar message [ 1869.804763] Lustre: server umount lustre-OST0000 complete [ 1871.843111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1871.846377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.854864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1871.857887] Lustre: Skipped 1 previous similar message [ 1877.986801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.992160] Lustre: Skipped 3 previous similar messages [ 1886.176123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1886.287092] Lustre: server umount lustre-MDT0000 complete [ 1888.228791] LustreError: 75588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1888.246219] LustreError: 75588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1888.711186] LustreError: 74311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786445816 with bad export cookie 14611739031782204312 [ 1888.716347] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.719097] LustreError: 74311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1888.905673] Lustre: server umount lustre-MDT0001 complete [ 1895.117305] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1896.667665] Key type lgssc unregistered [ 1896.828633] LNet: 78105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.833606] LNetError: 78105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.842503] LNet: Removed LNI 192.168.201.138@tcp [ 1897.349577] Key type .llcrypt unregistered [ 1897.350805] Key type ._llcrypt unregistered [ 1911.577110] Key type ._llcrypt registered [ 1911.578209] Key type .llcrypt registered [ 1911.633482] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1912.146992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1912.167879] alg: No test for adler32 (adler32-zlib) [ 1913.183499] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 1913.356522] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1915.017751] Key type lgssc registered [ 1915.686324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1920.411099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1920.426970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.699420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.539494] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1929.186428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1929.538385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1932.275375] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1936.764342] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1939.994018] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1944.035848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1944.375608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1948.023096] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1952.551850] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1957.841656] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1958.379798] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1959.410658] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1959.419493] Lustre: Skipped 1 previous similar message [ 1961.731655] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1961.735546] Lustre: Skipped 3 previous similar messages [ 1966.024448] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1966.026734] Lustre: Skipped 7 previous similar messages [ 1974.146315] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1974.150793] Lustre: Skipped 13 previous similar messages [ 1990.348020] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1990.351469] Lustre: Skipped 27 previous similar messages [ 2016.475240] Lustre: 87576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2017.801531] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 06:59:05 (1786445945) [ 2018.627911] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2019.663367] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 06:59:07 (1786445947) [ 2023.394689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2023.396751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.402128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2026.465312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2026.469673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2027.777955] Lustre: server umount lustre-OST0000 complete [ 2031.585929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.590647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.608334] Lustre: Skipped 1 previous similar message [ 2031.614773] Lustre: Skipped 1 previous similar message [ 2036.333102] Lustre: server umount lustre-MDT0000 complete [ 2036.707749] LustreError: 84915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.716700] LustreError: 84915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2038.473337] LustreError: 79477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786445966 with bad export cookie 6746143918787618809 [ 2038.479083] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2038.483778] LustreError: 79477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2038.686683] Lustre: server umount lustre-MDT0001 complete [ 2045.150881] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2046.603274] Key type lgssc unregistered [ 2046.744522] LNet: 88825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.747762] LNetError: 88825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.758060] LNet: Removed LNI 192.168.201.138@tcp [ 2047.156562] Key type .llcrypt unregistered [ 2047.158112] Key type ._llcrypt unregistered [ 2049.270432] Key type ._llcrypt registered [ 2049.271772] Key type .llcrypt registered [ 2049.327511] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2051.516376] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules [ 2052.273278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2052.286214] alg: No test for adler32 (adler32-zlib) [ 2053.207723] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 2053.386623] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2055.041099] Key type lgssc registered [ 2055.835782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2056.633579] LustreError: 89986:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2067.481482] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2073.141648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2077.404162] LustreError: 89986:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2077.425942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.712787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.770787] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2086.285781] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 07:00:13 (1786446013) [ 2087.911892] LustreError: 91002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.920842] LustreError: 91002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2091.393367] LustreError: 91036:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2091.431543] LustreError: 91036:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2091.438077] LustreError: 91036:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [ 2091.846124] Lustre: server umount lustre-MDT0000 complete [ 2109.480289] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2114.807571] Key type lgssc unregistered [ 2115.340358] LNet: 92533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.400050] LNetError: 92533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.456900] LNet: Removed LNI 192.168.201.138@tcp [ 2118.168199] Key type .llcrypt unregistered [ 2118.183088] Key type ._llcrypt unregistered [ 2154.479651] Key type ._llcrypt registered [ 2154.481863] Key type .llcrypt registered [ 2154.575802] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2155.907310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2156.253101] alg: No test for adler32 (adler32-zlib) [ 2157.389891] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 2157.605159] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2159.312218] Key type lgssc registered [ 2160.718552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2168.921610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2168.939103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.370487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.648396] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2180.578923] LustreError: 93769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2180.603059] LustreError: 93769:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2185.701321] LustreError: 93768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2186.713492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.799714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2195.332711] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2205.141450] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2210.693765] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2218.357733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2218.919183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2226.049374] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2234.581819] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2295.850954] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 07:03:41 (1786446221) [ 2298.496282] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2300.847994] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 07:03:47 (1786446227) [ 2307.049822] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.055532] Lustre: Skipped 1 previous similar message [ 2307.057748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2307.064500] Lustre: Skipped 2 previous similar messages [ 2311.754996] Lustre: server umount lustre-MDT0000 complete [ 2312.169930] LustreError: 93769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2312.203403] LustreError: 93769:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2316.227026] LustreError: 93750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786446244 with bad export cookie 4168999677223273243 [ 2316.241815] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2316.245908] LustreError: 93750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2317.098738] Lustre: server umount lustre-MDT0001 complete [ 2332.624719] Lustre: server umount lustre-OST0000 complete [ 2345.070696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.566205] LustreError: 98084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.794051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.074862] LustreError: 98085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.507393] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2360.292523] LustreError: 98084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.320349] LustreError: 98084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2361.292915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2362.267385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2367.935972] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2376.579703] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2382.932875] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2398.635157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.903961] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2398.912933] LustreError: 98071:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2398.958915] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2398.966393] LustreError: 100169:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2398.979331] LustreError: 100169:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2398.987220] LustreError: 100169:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2399.064540] Lustre: server umount lustre-OST0000 complete [ 2399.066962] LustreError: 100169:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2409.216537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2409.438097] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2409.441328] LustreError: 98074:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2409.473485] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2409.481046] LustreError: 100633:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2409.484917] LustreError: 100633:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2409.488175] LustreError: 100633:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2409.534597] Lustre: server umount lustre-OST0000 complete [ 2409.537528] LustreError: 100633:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2421.594570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2421.933702] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2421.940315] Lustre: Found index 0 for lustre-OST0000, updating log [ 2421.948920] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2422.035495] Lustre: lustre-OST0000: new disk, initializing [ 2422.039059] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2422.047671] Lustre: 101170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2422.505526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.412832] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2434.933119] Lustre: Failing over lustre-OST0000 [ 2435.257500] Lustre: server umount lustre-OST0000 complete [ 2442.210397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.217379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2442.223506] Lustre: Skipped 2 previous similar messages [ 2442.235735] Lustre: Skipped 1 previous similar message [ 2446.169692] Lustre: server umount lustre-MDT0000 complete [ 2447.330933] LustreError: 98079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.357430] LustreError: 98079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2450.484869] LustreError: 101168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786446378 with bad export cookie 4168999677223275980 [ 2450.491690] LustreError: 101168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2450.496174] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2461.078787] Lustre: server umount lustre-MDT0001 complete [ 2470.503428] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 07:06:37 (1786446397) [ 2490.207582] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2502.317161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.737525] LustreError: 103516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.824118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.766052] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2517.666947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.437126] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2526.818281] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2534.570785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2535.277937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2535.282840] Lustre: Skipped 1 previous similar message [ 2535.293597] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2542.430992] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2543.467620] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2543.494967] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2543.498614] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2552.508571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2552.638164] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2552.742891] Lustre: lustre-OST0001: new disk, initializing [ 2552.748802] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2552.756291] Lustre: 105833:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2559.168382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2559.203585] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2559.379500] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2560.043938] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2569.114342] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2574.604494] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2583.975216] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.38@tcp (stopping) [ 2585.061688] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2585.063431] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.094024] Lustre: Skipped 1 previous similar message [ 2586.485774] Lustre: server umount lustre-MDT0001 complete [ 2589.097068] LustreError: 103512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.107875] LustreError: 103512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2598.018623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.111016] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2598.117819] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2598.172849] Lustre: 103505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2600.195486] LustreError: 107401:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2600.203357] LustreError: 107401:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2600.216289] LustreError: 103508:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2600.337251] LustreError: 107411:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.138@tcp: cfg command failed: rc = -17 [ 2600.355394] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2600.355394] [ 2604.215860] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2604.521959] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2614.649647] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 07:09:01 (1786446541) [ 2634.473645] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2655.695074] Lustre: 109450:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2681.885530] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2686.220647] Lustre: 110586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2777.572431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2777.584938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2777.616081] Lustre: Skipped 1 previous similar message [ 2777.625272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2777.627673] Lustre: Skipped 3 previous similar messages [ 2783.361472] Lustre: server umount lustre-MDT0000 complete [ 2783.720450] LustreError: 110775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2783.746188] LustreError: 110775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 2787.384279] LustreError: 103498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786446715 with bad export cookie 4168999677223276988 [ 2787.396873] LustreError: 103498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2787.401037] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2787.929610] Lustre: server umount lustre-MDT0001 complete [ 2802.543936] Lustre: server umount lustre-OST0000 complete [ 2805.222853] Lustre: 93079:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786446717/real 1786446717] req@ffff89110200f800 x1873224490401152/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786446733 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2805.249755] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2805.262454] Lustre: Skipped 3 previous similar messages [ 2806.585995] Lustre: server umount lustre-OST0001 complete [ 2813.071350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2822.985812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2831.097841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2841.899068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2862.417771] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2873.882702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2873.897832] Lustre: lustre-MDT0000: reset Object Index mappings [ 2874.402630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2874.410979] Lustre: Skipped 2 previous similar messages [ 2879.119047] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2889.016783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.046162] Lustre: lustre-MDT0001: reset Object Index mappings [ 2894.008372] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2897.074872] Lustre: 115685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2903.728895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2910.535308] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2914.800655] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 2918.918306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 2919.042957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2925.064105] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2925.095517] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 2927.081841] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2934.934942] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2938.328678] Lustre: 117564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2988.034881] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 07:15:14 (1786446914) [ 3017.188973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3017.193224] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.198412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3017.198421] Lustre: Skipped 4 previous similar messages [ 3017.252843] Lustre: Skipped 3 previous similar messages [ 3019.448708] Lustre: server umount lustre-MDT0000 complete [ 3023.096307] LustreError: 117566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786446951 with bad export cookie 4168999677223323860 [ 3023.133575] LustreError: 117566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3023.139313] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3027.430664] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3027.449883] Lustre: Skipped 4 previous similar messages [ 3029.855378] Lustre: server umount lustre-MDT0001 complete [ 3040.191141] Lustre: server umount lustre-OST0000 complete [ 3044.046663] LustreError: 118521:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000401:0x78:0x0].0x0 (ffff89121ee4f100) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3050.351912] Lustre: server umount lustre-OST0001 complete [ 3057.445833] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 3068.196484] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3111.427983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3111.625786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3111.645307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3111.715571] Lustre: lustre-MDT0000: new disk, initializing [ 3111.798755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3111.801468] Lustre: Skipped 3 previous similar messages [ 3111.847443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3116.578818] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3123.975289] Lustre: server umount lustre-MDT0000 complete [ 3133.973565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.428782] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3134.471318] Lustre: 121829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3134.496210] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3139.480946] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3144.644800] Lustre: server umount lustre-MDT0000 complete [ 3153.555290] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 07:18:00 (1786447080) [ 3154.946890] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3156.777542] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 07:18:03 (1786447083) [ 3175.465795] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3186.024965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.883572] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3201.092229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3201.211215] Lustre: 123878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3201.267373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3201.272994] Lustre: Skipped 1 previous similar message [ 3201.406675] Lustre: lustre-MDT0001: new disk, initializing [ 3201.561326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3201.575289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3206.291284] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3211.607893] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3219.275258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3219.699188] Lustre: lustre-OST0000: new disk, initializing [ 3219.703584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3219.711644] Lustre: 125458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3221.479863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3221.499933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3221.665310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3227.258906] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3237.784128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3237.985495] Lustre: lustre-OST0001: new disk, initializing [ 3237.991596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3238.000396] Lustre: 126329:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3240.016536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3240.026669] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3240.161897] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3244.137710] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3253.750415] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3257.239873] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3612.209862] Lustre: 132742:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3612.215503] Lustre: 132742:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3612.220799] Lustre: 132742:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3612.226621] Lustre: 132742:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3612.235301] Lustre: 132742:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3612.243949] Lustre: 132742:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3825.918852] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:29:12 (1786447752) [ 3841.508545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3841.514255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3841.521141] Lustre: Skipped 2 previous similar messages [ 3841.524696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3841.528453] Lustre: Skipped 2 previous similar messages [ 3844.596424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3844.596873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3844.610338] Lustre: Skipped 1 previous similar message [ 3844.618158] Lustre: Skipped 3 previous similar messages [ 3846.012511] Lustre: server umount lustre-MDT0000 complete [ 3848.967149] LustreError: 132768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786447777 with bad export cookie 4168999677223391417 [ 3848.976057] LustreError: 132768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3848.977365] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3849.698127] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3849.700417] LustreError: 127140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3849.700822] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3849.713357] Lustre: Skipped 2 previous similar messages [ 3849.739704] LustreError: 127140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 3855.323362] Lustre: server umount lustre-MDT0001 complete [ 3864.369367] Lustre: server umount lustre-OST0000 complete [ 3872.850442] Lustre: server umount lustre-OST0001 complete [ 3877.311307] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 3885.129872] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3907.990719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3908.152129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3908.172118] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3908.238031] Lustre: lustre-MDT0000: new disk, initializing [ 3908.298745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3908.304822] Lustre: Skipped 5 previous similar messages [ 3908.316143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3910.961926] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3917.967709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.050117] Lustre: 135782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3918.107366] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3918.110357] Lustre: Skipped 1 previous similar message [ 3918.160597] Lustre: lustre-MDT0001: new disk, initializing [ 3918.278519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3918.293492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3921.413943] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3928.838294] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3932.814686] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3937.757302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3937.916530] Lustre: lustre-OST0000: new disk, initializing [ 3937.918447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3937.921466] Lustre: 137748:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3939.759593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3939.771213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3939.886870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3942.008744] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3948.709985] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3955.666831] Lustre: lustre-MDD0000: changelog on [ 3957.691206] Lustre: lustre-MDD0001: changelog on [ 5191.275911] Lustre: lustre-MDD0001: changelog off [ 5192.279758] Lustre: 139380:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5192.766721] Lustre: lustre-MDD0000: changelog off [ 5199.332211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5199.332636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5199.339077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5199.340477] LustreError: Skipped 1 previous similar message [ 5199.342950] Lustre: Skipped 3 previous similar messages [ 5201.405796] Lustre: server umount lustre-OST0000 complete [ 5203.431057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5203.438545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5203.447637] Lustre: Skipped 1 previous similar message [ 5203.449950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5209.485267] Lustre: server umount lustre-MDT0000 complete [ 5209.569510] LustreError: 136675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5209.577874] LustreError: 136675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5211.648246] LustreError: 135774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786449139 with bad export cookie 4168999677224173604 [ 5211.651422] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5211.658818] LustreError: 135774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5211.880900] Lustre: server umount lustre-MDT0001 complete [ 5216.899321] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 5218.144663] Key type lgssc unregistered [ 5218.309483] LNet: 146781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5218.313763] LNetError: 146781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5218.326431] LNet: Removed LNI 192.168.201.138@tcp [ 5218.737219] Key type .llcrypt unregistered [ 5218.738797] Key type ._llcrypt unregistered [ 5228.077575] Key type ._llcrypt registered [ 5228.079142] Key type .llcrypt registered [ 5228.126978] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 5235.979892] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5236.449280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5236.467262] alg: No test for adler32 (adler32-zlib) [ 5237.357080] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 5237.463443] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 5239.056210] Key type lgssc registered [ 5239.550241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5254.713699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5254.727645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5255.845120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5255.865978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5255.904536] Lustre: lustre-MDT0000: new disk, initializing [ 5255.936568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5255.947234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5257.878842] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5263.166497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5263.218437] Lustre: 149954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5263.251243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5263.253545] Lustre: Skipped 1 previous similar message [ 5263.292751] Lustre: lustre-MDT0001: new disk, initializing [ 5263.337193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5263.358545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5263.362588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5265.328850] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5270.290638] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5272.694596] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5275.624753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5275.749162] Lustre: lustre-OST0000: new disk, initializing [ 5275.751217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5275.753969] Lustre: 151923:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5275.795551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5277.383663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5277.390965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5277.432267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5278.157763] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5282.604177] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5285.314153] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_module obdecho/obdecho [ 5286.206338] LustreError: 152928:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5286.209697] LustreError: 152928:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5287.076526] LustreError: 153025:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5293.026377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5293.027336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5293.032183] Lustre: Skipped 1 previous similar message [ 5297.436477] Lustre: server umount lustre-MDT0000 complete [ 5298.884427] LustreError: 150752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786449227 with bad export cookie 17235465993567164118 [ 5298.888077] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5298.889305] LustreError: 150752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5299.029501] Lustre: server umount lustre-MDT0001 complete [ 5310.887692] Lustre: server umount lustre-OST0000 complete [ 5319.295820] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 5323.539443] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5339.096920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5339.228240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5339.243152] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5339.284125] Lustre: lustre-MDT0000: new disk, initializing [ 5339.323479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5339.342954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5341.614662] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5346.987361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5347.048071] Lustre: 156733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5347.079686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5347.082456] Lustre: Skipped 1 previous similar message [ 5347.120694] Lustre: lustre-MDT0001: new disk, initializing [ 5347.161370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5347.193232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5347.196873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5348.986302] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5353.950353] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5356.574678] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5359.756632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5359.904328] Lustre: lustre-OST0000: new disk, initializing [ 5359.907237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5359.910795] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5359.911816] Lustre: 158699:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5359.913015] Lustre: Skipped 2 previous similar messages [ 5359.946951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5362.561262] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5365.247378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5365.251931] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5365.270749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 5367.383522] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5369.462355] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5369.571951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5371.533821] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5371.653711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5375.458640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5375.461504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5375.466790] Lustre: Skipped 1 previous similar message [ 5375.471028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5376.484436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5376.484785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5378.757296] Lustre: server umount lustre-OST0000 complete [ 5381.603586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5381.613135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5381.615106] Lustre: Skipped 3 previous similar messages [ 5386.668241] Lustre: server umount lustre-MDT0000 complete [ 5386.721196] LustreError: 156743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5386.726882] LustreError: 156743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5388.468405] LustreError: 156724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786449316 with bad export cookie 17235465993567165077 [ 5388.473784] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5388.474109] LustreError: 156724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5388.636625] Lustre: server umount lustre-MDT0001 complete [ 5390.286519] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 07:55:18 (1786449318) [ 5391.024407] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5391.897458] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 07:55:19 (1786449319) [ 5395.413232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5395.597526] LustreError: 161105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5395.636376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5397.309569] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5401.037276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5401.058881] LustreError: 161106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5402.962223] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5406.224419] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5409.013795] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5412.256614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5415.039602] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5418.320450] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5434.270789] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) [ 5434.976758] Lustre: server umount lustre-MDT0000 complete [ 5436.689725] LustreError: 163603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786449364 with bad export cookie 17235465993567165959 [ 5436.694510] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5436.695376] LustreError: 163603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5436.900319] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5436.911262] Lustre: Skipped 2 previous similar messages [ 5436.913410] LustreError: 162343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5442.017045] LustreError: 161101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5442.022144] LustreError: 161101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5442.883369] Lustre: server umount lustre-MDT0001 complete [ 5445.755573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5451.337984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5451.443790] LustreError: 164961:0:(llog.c:1655:llog_backup()) MGC192.168.201.138@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5451.448768] Lustre: 164961:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.138@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5459.924410] LustreError: 164966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.240484] LustreError: 164966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.248753] LustreError: 164966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5511.056519] LustreError: 164968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5511.064849] LustreError: 164968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 5512.672336] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5512.674676] Lustre: 164961:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.138@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5553.127277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5553.131015] Lustre: Skipped 2 previous similar messages [ 5554.704545] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5556.196977] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5558.440193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5575.572513] LustreError: 164966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5575.583684] LustreError: 164966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 5581.797874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 5581.801912] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xef30ad00f4e57736 [ 5581.806187] Lustre: MGC192.168.201.138@tcp: Connection restored to 0@lo (at 0@lo) [ 5583.697179] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5586.896966] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5586.918704] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5587.521462] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5589.856738] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5592.534724] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5602.276119] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5602.277487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5602.277790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5602.277795] Lustre: Skipped 3 previous similar messages [ 5602.280383] Lustre: Skipped 1 previous similar message [ 5602.890725] Lustre: server umount lustre-MDT0000 complete [ 5604.242141] LustreError: 161086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786449532 with bad export cookie 17235465993567172406 [ 5604.242606] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5604.245579] LustreError: 161086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5604.352179] Lustre: server umount lustre-MDT0001 complete [ 5616.163181] Lustre: server umount lustre-OST0000 complete [ 5624.294496] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 5628.376854] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5642.443803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5642.527951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5642.540835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5642.581059] Lustre: lustre-MDT0000: new disk, initializing [ 5642.603184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5642.605430] Lustre: Skipped 1 previous similar message [ 5642.610549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5644.057513] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5648.446791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5648.487384] Lustre: 170038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5648.585169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5650.004109] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5654.036788] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5655.982990] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5658.463811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5658.580956] Lustre: 172004:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5660.482726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5660.527850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5660.643690] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5664.515098] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5666.118635] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5666.194141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5667.670267] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5667.744386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5670.883902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5670.884796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5670.887943] Lustre: Skipped 2 previous similar messages [ 5670.890138] Lustre: Skipped 2 previous similar messages [ 5674.508544] Lustre: server umount lustre-OST0000 complete [ 5681.821390] Lustre: server umount lustre-MDT0000 complete [ 5683.275433] LustreError: 170031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786449611 with bad export cookie 17235465993567189234 [ 5683.278298] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5683.281080] LustreError: 170031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5683.406181] Lustre: server umount lustre-MDT0001 complete [ 5686.661491] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:00:14 (1786449614) [ 5689.221029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.108808] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5694.449634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5696.057878] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5698.867988] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5700.919626] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5703.433183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5705.767901] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5708.398212] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5722.896143] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:00:50 (1786449650) [ 5724.131532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5724.131780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5724.135153] Lustre: Skipped 2 previous similar messages [ 5724.138423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5724.141124] Lustre: Skipped 6 previous similar messages [ 5730.007607] Lustre: server umount lustre-OST0000 complete [ 5737.507334] Lustre: server umount lustre-MDT0000 complete [ 5738.859275] LustreError: 174394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786449667 with bad export cookie 17235465993567190102 [ 5738.863088] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5738.863247] LustreError: 174394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5739.004549] Lustre: server umount lustre-MDT0001 complete [ 5742.853414] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 5743.915838] Key type lgssc unregistered [ 5744.053397] LNet: 178174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5744.055684] LNetError: 178174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5744.068310] LNet: Removed LNI 192.168.201.138@tcp [ 5744.398762] Key type .llcrypt unregistered [ 5744.400116] Key type ._llcrypt unregistered [ 5752.619338] Key type ._llcrypt registered [ 5752.620264] Key type .llcrypt registered [ 5752.663268] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5753.035767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5753.061936] alg: No test for adler32 (adler32-zlib) [ 5753.922173] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 5754.015509] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 5755.600160] Key type lgssc registered [ 5755.991477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5758.912202] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5758.919558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5774.368333] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5779.488329] LustreError: 179394:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.138@tcp: failed processing log, type 4: rc = -110 [ 5784.608399] LustreError: 179394:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5784.611736] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5784.613422] LustreError: MGC192.168.201.138@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5784.617020] LustreError: 179394:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5784.620138] LustreError: 179394:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 5784.622853] LustreError: 179394:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5784.643123] Lustre: server umount lustre-OST0000 complete [ 5784.644807] LustreError: 179394:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5787.588882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.732857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5790.244119] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5793.225137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5793.347050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5794.788569] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5797.417177] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5799.346146] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5801.834829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5801.914333] LustreError: 181705:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5801.917657] Lustre: 181705:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.138@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5801.961656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5804.062855] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5806.749973] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5808.098401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5808.100094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5808.102126] Lustre: Skipped 2 previous similar messages [ 5808.104157] Lustre: Skipped 1 previous similar message [ 5813.217814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5813.220374] Lustre: Skipped 3 previous similar messages [ 5813.659582] Lustre: server umount lustre-MDT0000 complete [ 5814.967233] LustreError: 179874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786449743 with bad export cookie 10209538006464567909 [ 5814.969550] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5814.971396] LustreError: 179874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5815.090059] Lustre: server umount lustre-MDT0001 complete [ 5826.649769] Lustre: server umount lustre-OST0000 complete [ 5829.938512] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 5833.787406] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5846.866058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5846.958201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5846.970057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5847.007279] Lustre: lustre-MDT0000: new disk, initializing [ 5847.027934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5847.034888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5848.455088] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5852.960232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.005290] Lustre: 185222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5853.033294] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5853.036268] Lustre: Skipped 1 previous similar message [ 5853.071844] Lustre: lustre-MDT0001: new disk, initializing [ 5853.108540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5853.121935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5853.125661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5854.722845] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5858.958813] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5861.165497] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5863.816482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5863.921488] Lustre: lustre-OST0000: new disk, initializing [ 5863.923430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5863.925660] Lustre: 187191:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5863.953153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5866.150834] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5870.080291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5870.083901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5870.126119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5870.368503] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5871.892267] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5871.967920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5873.463503] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5873.543327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5875.169179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5875.172860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5875.175152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5875.179325] Lustre: Skipped 1 previous similar message [ 5876.300180] Lustre: server umount lustre-OST0000 complete [ 5878.752865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5878.757577] LustreError: Skipped 1 previous similar message [ 5878.760179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5878.767996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5878.769960] Lustre: Skipped 1 previous similar message [ 5883.736582] Lustre: server umount lustre-MDT0000 complete [ 5885.052167] LustreError: 187190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786449813 with bad export cookie 10209538006464568609 [ 5885.055200] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5885.060652] LustreError: 187190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5885.179696] Lustre: server umount lustre-MDT0001 complete [ 5888.359647] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 08:03:36 (1786449816) [ 5894.755754] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 5895.980268] Key type lgssc unregistered [ 5896.113484] LNet: 190133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5896.116559] LNetError: 190133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5896.124647] LNet: Removed LNI 192.168.201.138@tcp [ 5896.454414] Key type .llcrypt unregistered [ 5896.456074] Key type ._llcrypt unregistered [ 5903.905598] Key type ._llcrypt registered [ 5903.906661] Key type .llcrypt registered [ 5903.948811] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5904.354519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5904.365761] alg: No test for adler32 (adler32-zlib) [ 5905.230323] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 5905.331228] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 5906.928201] Key type lgssc registered [ 5907.301097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5910.054914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5910.062656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.210119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5912.710249] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5915.782682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5915.920149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5917.548437] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5920.201877] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5922.289639] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5925.229065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5925.304174] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5925.305934] LustreError: 193192:0:(llog.c:1696:llog_backup()) MGC192.168.201.138@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5925.310027] Lustre: 193192:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.138@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5925.365939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5927.475250] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5930.480893] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5931.490264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5931.490834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5931.491649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5931.499770] Lustre: Skipped 1 previous similar message [ 5936.609995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5936.612676] Lustre: Skipped 4 previous similar messages [ 5937.363263] Lustre: server umount lustre-MDT0000 complete [ 5938.781486] LustreError: 191310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786449866 with bad export cookie 16174141501589227441 [ 5938.784570] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5938.786426] LustreError: 191310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5938.905853] Lustre: server umount lustre-MDT0001 complete [ 5950.555053] Lustre: server umount lustre-OST0000 complete [ 5954.266655] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 5957.905447] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5970.913845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5971.020478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5971.034211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5971.074851] Lustre: lustre-MDT0000: new disk, initializing [ 5971.105492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5971.114430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5972.596475] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5977.280763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.325441] Lustre: 196710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5977.349346] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5977.350767] Lustre: Skipped 1 previous similar message [ 5977.352174] LustreError: 197494:0:(llog.c:1696:llog_backup()) MGC192.168.201.138@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5977.355353] LustreError: 197494:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5977.358232] Lustre: 197494:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.138@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5977.367075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5977.369365] Lustre: Skipped 1 previous similar message [ 5977.404348] Lustre: lustre-MDT0001: new disk, initializing [ 5977.437358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5977.449829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5977.454120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5978.902510] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5983.180379] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5985.306164] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5987.907799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5988.016845] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5988.018342] Lustre: Skipped 1 previous similar message [ 5988.019838] LustreError: 198657:0:(llog.c:1696:llog_backup()) MGC192.168.201.138@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5988.023411] LustreError: 198657:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5988.025789] Lustre: 198657:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.138@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5988.029287] Lustre: 198657:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5988.039046] Lustre: lustre-OST0000: new disk, initializing [ 5988.041144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5988.044560] Lustre: 198681:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5988.080270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5990.382774] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5991.165689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5991.169971] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5991.208509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5994.586235] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5996.140296] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5996.215717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5997.764568] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5997.846092] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6001.634060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6001.636463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6001.639443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6001.643858] Lustre: Skipped 2 previous similar messages [ 6004.684818] Lustre: server umount lustre-OST0000 complete [ 6006.753107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.755661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6006.758968] Lustre: Skipped 1 previous similar message [ 6006.761478] Lustre: Skipped 1 previous similar message [ 6011.873295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6011.876380] Lustre: Skipped 1 previous similar message [ 6012.190167] Lustre: server umount lustre-MDT0000 complete [ 6013.726198] LustreError: 196748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786449941 with bad export cookie 16174141501589228141 [ 6013.728742] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6013.730832] LustreError: 196748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6013.852987] Lustre: server umount lustre-MDT0001 complete [ 6017.621843] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 08:05:45 (1786449945) [ 6024.272572] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6028.093265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6028.229864] LustreError: 201802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6028.235142] LustreError: 201802:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6028.251390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6029.777564] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6032.876720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6034.462960] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6035.524987] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6037.926239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6040.331160] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6043.438280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6043.515722] Lustre: lustre-OST0001: new disk, initializing [ 6043.517931] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6043.521734] Lustre: 204112:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6045.750001] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6048.745770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6048.749131] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6048.763698] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6051.513319] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6056.395600] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6057.080651] Lustre: DEBUG MARKER: ADD OST3 [ 6058.928909] loop: module loaded [ 6058.950491] loop0: detected capacity change from 0 to 204800000 [ 6059.052261] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6059.058714] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6059.064272] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6061.389079] loop0: detected capacity change from 0 to 204800000 [ 6063.384702] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6063.453192] Lustre: lustre-OST0003: new disk, initializing [ 6063.455687] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6063.458207] Lustre: 205841:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6063.496741] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6063.498496] Lustre: Skipped 3 previous similar messages [ 6064.120225] Lustre: DEBUG MARKER: STOP OST3 [ 6064.823842] Lustre: Failing over lustre-OST0003 [ 6064.870072] Lustre: server umount lustre-OST0003 complete [ 6065.163794] LustreError: 203286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6065.836599] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6069.580672] Lustre: DEBUG MARKER: START OST3 again [ 6070.172386] LustreError: 206232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6070.178124] LustreError: 206232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6072.061355] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6072.152296] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 6073.619562] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 6073.628387] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 6074.210300] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6077.420289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6077.423384] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6077.432803] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 6087.905578] Lustre: Failing over lustre-OST0003 [ 6087.949305] Lustre: server umount lustre-OST0003 complete [ 6092.772787] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6092.774618] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6092.775105] LustreError: Skipped 2 previous similar messages [ 6092.779898] LustreError: 203287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6092.785196] LustreError: 203287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6097.892451] LustreError: 203288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6097.896861] LustreError: 203288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6103.011954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6103.013254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6103.016099] Lustre: Skipped 4 previous similar messages [ 6103.018641] Lustre: Skipped 2 previous similar messages [ 6106.796262] Lustre: server umount lustre-MDT0000 complete [ 6108.094375] LustreError: 201784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786450036 with bad export cookie 16174141501589229023 [ 6108.100257] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6108.101391] LustreError: 201784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6108.130428] LustreError: 202537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6108.139944] LustreError: 202537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6114.356640] Lustre: server umount lustre-MDT0001 complete [ 6115.822566] Lustre: server umount lustre-OST0000 complete [ 6117.254947] Lustre: server umount lustre-OST0001 complete [ 6119.481821] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 6122.314355] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6135.193075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6135.278264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6135.288171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6135.320660] Lustre: lustre-MDT0000: new disk, initializing [ 6135.339603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6135.341218] Lustre: Skipped 1 previous similar message [ 6135.345647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6136.785712] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6141.027168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6141.080705] Lustre: 211251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6141.181285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6142.590681] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6146.355236] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6148.277320] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6150.640950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6150.749222] Lustre: 213219:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6152.065548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6152.068467] Lustre: Skipped 1 previous similar message [ 6152.070848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6152.114818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6152.784790] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6156.535188] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6157.967699] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6158.054952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6159.386171] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6159.458201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6162.403564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.403605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6162.407767] Lustre: Skipped 2 previous similar messages [ 6162.410160] LustreError: Skipped 1 previous similar message [ 6162.411166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6162.416106] Lustre: Skipped 6 previous similar messages [ 6166.225402] Lustre: server umount lustre-OST0000 complete [ 6172.128934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6173.657116] Lustre: server umount lustre-MDT0000 complete [ 6175.006648] LustreError: 213217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786450103 with bad export cookie 16174141501589231291 [ 6175.009742] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6175.011565] LustreError: 213217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6175.147986] Lustre: server umount lustre-MDT0001 complete [ 6176.426912] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 08:08:24 (1786450104) [ 6182.372579] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 6186.062505] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6198.994191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6199.081179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6199.094811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6199.096758] Lustre: Skipped 3 previous similar messages [ 6199.132601] Lustre: lustre-MDT0000: new disk, initializing [ 6199.133980] Lustre: Skipped 2 previous similar messages [ 6199.156361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6200.575037] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6205.123687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6205.166703] Lustre: 217816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6205.268034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6206.771867] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6210.797759] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6212.954293] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6215.459266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6215.563983] Lustre: 219784:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6217.004084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6217.684284] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6221.739019] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6223.220209] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6223.298371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6224.745291] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6224.821164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6227.427122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6227.431182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6227.433450] LustreError: Skipped 1 previous similar message [ 6227.433943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6227.433952] Lustre: Skipped 3 previous similar messages [ 6227.440410] Lustre: Skipped 3 previous similar messages [ 6231.572570] Lustre: server umount lustre-OST0000 complete [ 6247.392108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6247.465975] Lustre: server umount lustre-MDT0000 complete [ 6248.772473] LustreError: 217809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786450176 with bad export cookie 16174141501589232173 [ 6248.774862] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6248.776366] LustreError: 217809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6248.900277] Lustre: server umount lustre-MDT0001 complete [ 6251.768500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6251.916802] LustreError: 222000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6251.923175] LustreError: 222000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6253.319799] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6256.202355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6257.694338] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6260.059865] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6261.981135] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6264.340027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6264.449152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6264.451423] Lustre: Skipped 7 previous similar messages [ 6266.417908] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6268.708384] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6287.329502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6287.332150] LustreError: Skipped 1 previous similar message [ 6287.333672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6287.337636] Lustre: Skipped 2 previous similar messages [ 6291.536533] Lustre: server umount lustre-OST0000 complete [ 6293.474586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6293.476964] Lustre: Skipped 12 previous similar messages [ 6307.296121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6307.358260] Lustre: server umount lustre-MDT0000 complete [ 6308.702351] LustreError: 221982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786450236 with bad export cookie 16174141501589233055 [ 6308.706842] LustreError: 221982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6308.706949] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6308.833491] Lustre: server umount lustre-MDT0001 complete [ 6400.850814] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6401.925752] Key type lgssc unregistered [ 6402.066345] LNet: 225687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6402.069281] LNetError: 225687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6402.077353] LNet: Removed LNI 192.168.201.138@tcp [ 6402.401101] Key type .llcrypt unregistered [ 6402.401949] Key type ._llcrypt unregistered [ 6410.634550] Key type ._llcrypt registered [ 6410.635976] Key type .llcrypt registered [ 6410.672383] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 6416.733493] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6417.031714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6417.039111] alg: No test for adler32 (adler32-zlib) [ 6417.892952] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 6417.976715] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6419.560113] Key type lgssc registered [ 6419.919331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6431.517076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6431.523047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6432.603856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6432.616417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6432.650515] Lustre: lustre-MDT0000: new disk, initializing [ 6432.673848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6432.680951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6434.063864] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6438.331528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6438.370135] Lustre: 229012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6438.391528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6438.392925] Lustre: Skipped 1 previous similar message [ 6438.427037] Lustre: lustre-MDT0001: new disk, initializing [ 6438.457616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6438.468096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6438.472631] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6439.828740] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6443.652343] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6445.548853] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6447.727831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6447.828647] Lustre: lustre-OST0000: new disk, initializing [ 6447.830540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6447.833203] Lustre: 230980:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6447.859311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6449.804170] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6453.241821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6453.245564] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6453.274833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6453.581402] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6454.978129] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6455.055671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6456.447490] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6456.533865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6458.338648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6458.339146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6458.340430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6458.340956] LustreError: Skipped 1 previous similar message [ 6458.347563] Lustre: Skipped 1 previous similar message [ 6459.215551] Lustre: server umount lustre-OST0000 complete [ 6463.457584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6463.458367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6463.461836] Lustre: Skipped 1 previous similar message [ 6463.464010] Lustre: Skipped 1 previous similar message [ 6466.590317] Lustre: server umount lustre-MDT0000 complete [ 6467.900355] LustreError: 229005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786450396 with bad export cookie 7343788331481089171 [ 6467.903448] LustreError: 229005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6467.903660] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6468.016788] Lustre: server umount lustre-MDT0001 complete [ 6474.086283] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6477.678745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6477.852200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6479.271603] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6482.319431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6482.438493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6483.864849] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6484.904338] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6487.321510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6489.562172] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6492.590813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6492.672883] Lustre: lustre-OST0001: new disk, initializing [ 6492.675198] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6492.678293] Lustre: 236210:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6492.716620] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6492.719189] Lustre: Skipped 1 previous similar message [ 6494.814204] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6499.842259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6499.845125] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6499.885757] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6500.361541] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6507.101872] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6511.218954] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 08:13:59 (1786450439) [ 6513.121183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6513.124174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6513.129934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6513.132582] Lustre: Skipped 1 previous similar message [ 6515.171473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6515.173777] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6515.179269] Lustre: Skipped 2 previous similar messages [ 6518.049403] Lustre: server umount lustre-MDT0000 complete [ 6519.310392] LustreError: 235039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786450447 with bad export cookie 7343788331481090039 [ 6519.313506] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6519.314414] LustreError: 235039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6519.437795] Lustre: server umount lustre-MDT0001 complete [ 6529.930841] Lustre: server umount lustre-OST0000 complete [ 6541.621591] Lustre: server umount lustre-OST0001 complete [ 6544.237417] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 6547.959692] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6560.801209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6560.888051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6560.899172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6560.933276] Lustre: lustre-MDT0000: new disk, initializing [ 6560.957855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.965352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6562.363687] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6566.701324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6566.742427] Lustre: 240154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6566.844972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6568.286713] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6572.669159] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6574.866676] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6577.455163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6577.576845] Lustre: lustre-OST0000: new disk, initializing [ 6577.578351] Lustre: Skipped 1 previous similar message [ 6577.579830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6577.581580] Lustre: Skipped 2 previous similar messages [ 6577.583938] Lustre: 242126:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6578.704472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6578.707569] Lustre: Skipped 1 previous similar message [ 6578.714821] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6578.765432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6579.752616] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6583.797556] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6585.345322] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6585.424972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6586.929599] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6587.004487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6588.900364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6588.901168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6588.905474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6588.908528] Lustre: Skipped 4 previous similar messages [ 6593.744122] Lustre: server umount lustre-OST0000 complete [ 6597.601535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6597.604608] LustreError: Skipped 1 previous similar message [ 6597.605839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6597.609577] Lustre: Skipped 1 previous similar message [ 6597.611558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6609.376133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6609.442847] Lustre: server umount lustre-MDT0000 complete [ 6610.788544] LustreError: 240950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786450538 with bad export cookie 7343788331481092160 [ 6610.791862] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6610.792719] LustreError: 240950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6610.925607] Lustre: server umount lustre-MDT0001 complete [ 6613.989312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6614.143206] LustreError: 244340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6614.149786] LustreError: 244340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6614.168625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6614.170326] Lustre: Skipped 2 previous similar messages [ 6615.695929] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6618.565544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.077251] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6622.567884] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6624.519331] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6626.817481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6628.809430] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6631.237118] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6776.800986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6776.803147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6776.806098] Lustre: Skipped 1 previous similar message [ 6776.807639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6776.810256] Lustre: Skipped 6 previous similar messages [ 6780.687920] Lustre: server umount lustre-OST0000 complete [ 6788.184783] Lustre: server umount lustre-MDT0000 complete [ 6789.372117] LustreError: 246181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786450717 with bad export cookie 7343788331481093042 [ 6789.375051] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6789.375990] LustreError: 246181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6789.498975] Lustre: server umount lustre-MDT0001 complete [ 6903.816668] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6904.730563] Key type lgssc unregistered [ 6904.846316] LNet: 247949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6904.848878] LNetError: 247949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6904.858257] LNet: Removed LNI 192.168.201.138@tcp [ 6905.153087] Key type .llcrypt unregistered [ 6905.154036] Key type ._llcrypt unregistered [ 6912.653714] Key type ._llcrypt registered [ 6912.655021] Key type .llcrypt registered [ 6912.695251] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 6918.745036] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6919.122058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6919.136029] alg: No test for adler32 (adler32-zlib) [ 6919.984926] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 6920.078606] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6921.656152] Key type lgssc registered [ 6921.995242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6936.328290] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6939.861346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6939.866499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6940.941220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6940.951320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6940.984607] Lustre: lustre-MDT0000: new disk, initializing [ 6941.003229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6941.009555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6942.321894] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6946.393051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6946.431269] Lustre: 251987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6946.456145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6946.458449] Lustre: Skipped 1 previous similar message [ 6946.496232] Lustre: lustre-MDT0001: new disk, initializing [ 6946.531644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6946.546360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6946.551789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6947.883708] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6950.109627] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6952.273548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6952.371268] Lustre: lustre-OST0000: new disk, initializing [ 6952.372515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6952.374414] Lustre: 253618:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6952.405129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6954.247265] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6958.072924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6958.076562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6958.119346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6958.271770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6958.339194] Lustre: lustre-OST0001: new disk, initializing [ 6958.342083] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6958.345269] Lustre: 254491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6958.380845] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6960.218360] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6965.467055] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6967.883368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6967.886224] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6967.913359] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6973.105933] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6977.504886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6977.507018] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6977.510470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6978.529198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6978.531272] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6978.535414] Lustre: Skipped 2 previous similar messages [ 6982.191183] Lustre: server umount lustre-MDT0000 complete [ 6983.421378] LustreError: 251981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786450911 with bad export cookie 7292324015005319756 [ 6983.423544] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6983.425048] LustreError: 251981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6983.543715] Lustre: server umount lustre-MDT0001 complete [ 6993.989268] Lustre: server umount lustre-OST0000 complete [ 7005.474584] Lustre: server umount lustre-OST0001 complete [ 7009.746371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7012.653399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7015.363199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7018.705342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7025.135164] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7028.524918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7028.531365] Lustre: lustre-MDT0000: reset Object Index mappings [ 7028.689331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7030.044583] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7032.976881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7032.985435] Lustre: lustre-MDT0001: reset Object Index mappings [ 7034.582697] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7035.600169] Lustre: 260621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7037.818420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7037.973056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7037.975665] Lustre: Skipped 1 previous similar message [ 7039.014842] LustreError: 260976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7039.018057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7039.020535] LustreError: 260976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7040.021992] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7042.705767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7043.829280] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7044.740088] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7047.900978] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7049.093692] Lustre: 262488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7053.793138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7053.796764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7053.802673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7053.805010] Lustre: Skipped 3 previous similar messages [ 7056.416275] Lustre: server umount lustre-MDT0000 complete [ 7057.624157] LustreError: 259462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786450985 with bad export cookie 7292324015005326434 [ 7057.627484] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7057.627523] LustreError: 259462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7057.748204] Lustre: server umount lustre-MDT0001 complete [ 7069.632769] Lustre: server umount lustre-OST0000 complete [ 7080.027311] Lustre: server umount lustre-OST0001 complete [ 7085.991295] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 08:23:34 (1786451014) [ 7090.853423] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 7093.699468] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7104.489568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7104.568972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7104.578508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7104.614110] Lustre: lustre-MDT0000: new disk, initializing [ 7104.633161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7104.635679] Lustre: Skipped 1 previous similar message [ 7104.643324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7105.958960] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7110.146070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7110.185246] Lustre: 266735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7110.283300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7111.584709] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7115.154214] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7116.886187] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7119.017994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7119.120517] Lustre: lustre-OST0000: new disk, initializing [ 7119.122456] Lustre: Skipped 1 previous similar message [ 7119.124602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7119.127457] Lustre: Skipped 2 previous similar messages [ 7119.130225] Lustre: 268701:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7120.762715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7120.765617] Lustre: Skipped 1 previous similar message [ 7120.767788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7120.798056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7120.974464] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7124.503857] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7125.938984] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7126.010933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7127.341320] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7127.412900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7131.107290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7131.107499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7131.111329] Lustre: Skipped 3 previous similar messages [ 7131.113689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7131.116520] Lustre: Skipped 5 previous similar messages [ 7134.166326] Lustre: server umount lustre-OST0000 complete [ 7136.225983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7136.226117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7136.226871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7136.231218] Lustre: Skipped 1 previous similar message [ 7136.234191] LustreError: Skipped 1 previous similar message [ 7141.534456] Lustre: server umount lustre-MDT0000 complete [ 7142.679121] LustreError: 267529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786451070 with bad export cookie 7292324015005328506 [ 7142.682729] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7142.683663] LustreError: 267529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7142.791154] Lustre: server umount lustre-MDT0001 complete [ 7148.631192] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7151.852854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7151.972239] LustreError: 271629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7151.986487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7151.989527] Lustre: Skipped 2 previous similar messages [ 7153.237541] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7155.907433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7157.370788] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7158.341904] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7160.518548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7162.460593] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7165.184484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7165.253547] Lustre: lustre-OST0001: new disk, initializing [ 7165.255099] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7165.257783] Lustre: 273933:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7166.345736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7166.349631] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7166.382631] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7167.110523] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7171.510199] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7172.679346] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7174.436663] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7174.441414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7176.679569] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.680519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7176.681127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7176.684154] Lustre: Skipped 3 previous similar messages [ 7176.685451] Lustre: Skipped 5 previous similar messages [ 7182.307206] Lustre: server umount lustre-MDT0000 complete [ 7183.514168] LustreError: 272363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786451111 with bad export cookie 7292324015005329388 [ 7183.517591] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7183.517882] LustreError: 272363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7183.628302] Lustre: server umount lustre-MDT0001 complete [ 7195.155890] Lustre: server umount lustre-OST0000 complete [ 7206.632398] Lustre: server umount lustre-OST0001 complete [ 7211.812141] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7214.957641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7215.072784] LustreError: 276981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7216.436434] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7219.238704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7219.372119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7219.374568] Lustre: Skipped 4 previous similar messages [ 7220.703233] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7221.617332] Lustre: 278121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7223.758571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7224.931839] LustreError: 278476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.936251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7225.770248] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7228.592882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7229.678555] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7230.517704] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7233.578448] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7234.753676] Lustre: 279996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7236.588485] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7236.593293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7239.774257] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 08:26:07 (1786451167) [ 7255.524718] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7255.526040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7255.527776] Lustre: Skipped 2 previous similar messages [ 7255.528322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7255.529394] Lustre: Skipped 8 previous similar messages [ 7257.314045] Lustre: server umount lustre-MDT0000 complete [ 7258.422271] LustreError: 279999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786451186 with bad export cookie 7292324015005335681 [ 7258.425863] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7258.426137] LustreError: 279999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7260.640981] LustreError: 279469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7260.644933] LustreError: 279469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7264.619996] Lustre: server umount lustre-MDT0001 complete [ 7280.096094] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7280.130543] Lustre: server umount lustre-OST0000 complete [ 7287.482250] Lustre: server umount lustre-OST0001 complete [ 7289.520227] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 7292.665783] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7306.927838] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7310.066107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7310.135762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7310.143217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7310.174219] Lustre: lustre-MDT0000: new disk, initializing [ 7310.197066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7311.411468] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7315.484858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7315.526112] Lustre: 284228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7315.617095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7316.913608] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7321.182845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7321.273218] Lustre: 285858:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7322.156283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7323.031564] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7326.880312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7328.635180] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7328.928361] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7332.929228] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7337.877409] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 08:27:45 (1786451265) [ 7339.490896] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7339.493967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7339.499526] Lustre: Skipped 6 previous similar messages [ 7339.502522] Lustre: Skipped 6 previous similar messages [ 7344.738386] Lustre: server umount lustre-MDT0000 complete [ 7345.922097] LustreError: 284220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786451274 with bad export cookie 7292324015005343570 [ 7345.925546] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7345.928285] LustreError: 284220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7346.029206] Lustre: server umount lustre-MDT0001 complete [ 7356.931295] Lustre: server umount lustre-OST0000 complete [ 7368.352163] Lustre: server umount lustre-OST0001 complete [ 7370.969174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7371.114515] LustreError: 288974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7371.120282] LustreError: 288974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7371.144770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7371.147406] Lustre: Skipped 6 previous similar messages [ 7372.431773] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7375.181733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7376.640358] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7378.888706] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7380.522602] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7382.655859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7384.605533] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7386.923257] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7390.180625] LustreError: 290805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7390.188176] LustreError: 290805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7403.026350] loop1: detected capacity change from 0 to 204800000 [ 7403.098590] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7403.102341] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7403.106315] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7404.694245] loop1: detected capacity change from 0 to 204800000 [ 7406.526211] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7406.553916] LustreError: 288963:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs9940: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7406.557492] LustreError: 288963:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs9940 [ 7406.560511] LustreError: 288963:0:(mgs_handler.c:517:mgs_target_reg()) fs9940-MDT0001: Error checking target index: rc = -13 [ 7406.565624] LustreError: fs9940-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7406.569602] LustreError: 292710:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 7406.572742] LustreError: 292710:0:(tgt_mount.c:2129:server_put_super()) no obd fs9940-MDT0001 [ 7406.575402] LustreError: 292710:0:(tgt_mount.c:116:server_deregister_mount()) fs9940-MDT0001 not registered [ 7406.587911] Lustre: server umount fs9940-MDT0001 complete [ 7406.589727] LustreError: 292710:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7409.224564] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7409.254521] Lustre: Setting parameter fs9940-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs9940-MDT0001 [ 7409.257455] Lustre: Skipped 2 previous similar messages [ 7409.274045] Lustre: srv-fs9940-MDT0001: No data found on store. Initialize space. [ 7409.275748] Lustre: Skipped 4 previous similar messages [ 7409.309319] Lustre: fs9940-MDT0001: new disk, initializing [ 7409.310771] Lustre: Skipped 3 previous similar messages [ 7409.343231] Lustre: srv-fs9940-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7409.346043] Lustre: 293176:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9940-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7409.349164] LustreError: 293176:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9940-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7410.557701] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7413.442726] Lustre: server umount fs9940-MDT0001 complete [ 7417.708574] loop1: detected capacity change from 0 to 204800000 [ 7419.553854] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7419.676973] Lustre: 294744:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9940-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7419.680105] LustreError: 294744:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9940-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7420.952220] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7422.428798] Lustre: server umount fs9940-MDT0001 complete [ 7422.864237] LustreError: 290803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7422.870868] LustreError: 290803:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 7426.528849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7426.531657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7426.536393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7426.538432] Lustre: Skipped 5 previous similar messages [ 7431.704653] Lustre: server umount lustre-MDT0000 complete [ 7434.713223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7434.755126] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7436.219503] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7438.752943] loop1: detected capacity change from 0 to 204800000 [ 7440.359732] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7440.677291] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7440.779707] Lustre: 297185:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9940-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7440.783035] LustreError: 297185:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9940-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7442.085606] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7443.878534] Lustre: server umount fs9940-MDT0001 complete [ 7456.478497] Lustre: server umount lustre-OST0000 complete [ 7460.834275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7460.837454] LustreError: Skipped 1 previous similar message [ 7472.096112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7472.162691] Lustre: server umount lustre-MDT0000 complete [ 7473.489043] LustreError: 289717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786451401 with bad export cookie 7292324015005346244 [ 7473.491670] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7473.493603] LustreError: 289717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7473.624973] Lustre: server umount lustre-MDT0001 complete [ 7477.217547] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 7478.222102] Key type lgssc unregistered [ 7478.354411] LNet: 299526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7478.357184] LNetError: 299526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7478.366358] LNet: Removed LNI 192.168.201.138@tcp [ 7478.671112] Key type .llcrypt unregistered [ 7478.672364] Key type ._llcrypt unregistered [ 7486.117635] Key type ._llcrypt registered [ 7486.118547] Key type .llcrypt registered [ 7486.153758] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7486.436402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7486.447635] alg: No test for adler32 (adler32-zlib) [ 7487.292955] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 7487.374564] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 7488.952107] Key type lgssc registered [ 7489.279253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7491.688284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7491.694133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.823288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7494.110822] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7496.919544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.052162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7498.377487] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7500.764208] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7502.561508] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7504.775379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7504.892587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7506.730165] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7506.984824] LustreError: 302602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7506.990543] LustreError: 302602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7509.187596] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7512.036356] LustreError: 302601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7517.156936] LustreError: 302603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7517.163656] LustreError: 302603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7519.639650] LustreError: 302997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7524.753922] LustreError: 302603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7524.760345] LustreError: 302603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7525.530516] loop1: detected capacity change from 0 to 204800000 [ 7525.586286] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7525.589726] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7525.593062] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7527.277107] loop1: detected capacity change from 0 to 204800000 [ 7529.070284] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7529.120974] Lustre: fs20762-OST0000: new disk, initializing [ 7529.122443] Lustre: srv-fs20762-OST0000: No data found on store. Initialize space. [ 7529.124582] Lustre: 304525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7529.145447] Lustre: fs20762-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7531.065134] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7534.463722] Lustre: server umount fs20762-OST0000 complete [ 7534.992913] LustreError: 302998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7535.000022] LustreError: 302998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7538.393491] loop1: detected capacity change from 0 to 204800000 [ 7540.332341] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7540.402082] Lustre: fs20762-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7542.252517] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7545.112038] loop2: detected capacity change from 0 to 204800000 [ 7545.188097] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7545.193311] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7545.197976] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7546.902296] loop2: detected capacity change from 0 to 204800000 [ 7548.730478] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7548.759391] LustreError: fs20762-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 7548.763240] LustreError: fs20762-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7548.766705] LustreError: 307567:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 7548.768662] LustreError: 307567:0:(tgt_mount.c:2129:server_put_super()) no obd fs20762-OST0001 [ 7548.770631] LustreError: 307567:0:(tgt_mount.c:116:server_deregister_mount()) fs20762-OST0001 not registered [ 7548.780675] Lustre: server umount fs20762-OST0001 complete [ 7548.781814] LustreError: 307567:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7552.067211] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7552.111165] Lustre: fs20762-OST0001: new disk, initializing [ 7552.112628] Lustre: srv-fs20762-OST0001: No data found on store. Initialize space. [ 7552.115070] Lustre: 308118:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7552.136812] Lustre: fs20762-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7552.997424] LustreError: 302998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7553.002287] LustreError: 302998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 7553.935891] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7557.317126] Lustre: server umount fs20762-OST0000 complete [ 7560.345872] Lustre: server umount fs20762-OST0001 complete [ 7564.150081] loop1: detected capacity change from 0 to 204800000 [ 7565.971914] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7567.822171] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7569.927161] loop2: detected capacity change from 0 to 204800000 [ 7571.735623] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7571.791533] Lustre: fs20762-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7571.793389] Lustre: Skipped 1 previous similar message [ 7573.689266] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7576.547956] Lustre: server umount fs20762-OST0000 complete [ 7579.608562] Lustre: server umount fs20762-OST0001 complete [ 7583.714182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7583.714935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7583.719953] LustreError: Skipped 1 previous similar message [ 7583.720623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7583.723925] Lustre: Skipped 1 previous similar message [ 7588.834345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7588.835889] LustreError: 302998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7588.836616] Lustre: Skipped 2 previous similar messages [ 7588.840931] LustreError: 302998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 7589.266592] Lustre: server umount lustre-OST0000 complete [ 7593.953652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.957461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7593.959592] Lustre: Skipped 1 previous similar message [ 7596.632986] Lustre: server umount lustre-MDT0000 complete [ 7597.852189] LustreError: 300757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786451526 with bad export cookie 10663339446324967040 [ 7597.855681] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7597.856464] LustreError: 300757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7597.981514] Lustre: server umount lustre-MDT0001 complete [ 7601.561865] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 7602.573096] Key type lgssc unregistered [ 7602.692380] LNet: 314029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7602.695885] LNetError: 314029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7602.703345] LNet: Removed LNI 192.168.201.138@tcp [ 7602.991086] Key type .llcrypt unregistered [ 7602.991915] Key type ._llcrypt unregistered [ 7612.703158] Key type ._llcrypt registered [ 7612.704638] Key type .llcrypt registered [ 7612.740927] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 7618.660948] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7618.961698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7618.998468] alg: No test for adler32 (adler32-zlib) [ 7619.849158] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 7619.931414] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 7621.504128] Key type lgssc registered [ 7621.844622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7633.333655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7633.340869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7634.448159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7634.461343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7634.505984] Lustre: lustre-MDT0000: new disk, initializing [ 7634.529433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7634.536517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7635.958977] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7640.127939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7640.168510] Lustre: 317858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7640.190198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7640.191695] Lustre: Skipped 1 previous similar message [ 7640.225721] Lustre: lustre-MDT0001: new disk, initializing [ 7640.256552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7640.267932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7640.271603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7641.596427] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7645.246883] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7647.073968] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7649.259762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7649.362162] Lustre: lustre-OST0000: new disk, initializing [ 7649.363581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7649.365724] Lustre: 319827:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7649.395058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7651.234441] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7654.774426] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7656.821890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7656.824484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7656.853988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7662.071538] Lustre: Mounted lustre-client - version 2.17.56_51_gd63417b [ 7662.379477] Lustre: Unmounted lustre-client [ 7667.171798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7667.173091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7667.174548] LustreError: Skipped 1 previous similar message [ 7667.179188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7667.182344] Lustre: Skipped 1 previous similar message [ 7671.318254] Lustre: server umount lustre-OST0000 complete [ 7676.384748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7676.387668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7676.392157] Lustre: Skipped 1 previous similar message [ 7676.394209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7677.409093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7677.409522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7678.743681] Lustre: server umount lustre-MDT0000 complete [ 7679.912897] LustreError: 319826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786451608 with bad export cookie 5977415834895026865 [ 7679.915120] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7679.916984] LustreError: 319826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7680.033066] Lustre: server umount lustre-MDT0001 complete [ 7683.390472] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 7684.310485] Key type lgssc unregistered [ 7712.096219] LNet: 1 local NIs in recovery (showing 1): 192.168.201.138@tcp [ 7712.098718] LNet: 321987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7712.100932] LNetError: 321987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7787.878384] LNet: Removed LNI 192.168.201.138@tcp [ 7788.248674] Key type .llcrypt unregistered [ 7788.250505] Key type ._llcrypt unregistered [ 7794.028806] Key type ._llcrypt registered [ 7794.029941] Key type .llcrypt registered [ 7794.065138] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 7800.282908] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7800.628796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7800.695820] alg: No test for adler32 (adler32-zlib) [ 7801.555086] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 7801.647704] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 7803.224103] Key type lgssc registered [ 7803.601622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7813.773198] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 08:35:41 (1786451741) [ 7816.136988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7816.142302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7817.222833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7817.232854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7817.271524] Lustre: lustre-MDT0000: new disk, initializing [ 7817.293424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7817.300715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7818.640896] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7822.779208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7822.819892] Lustre: 325184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7822.843514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7822.845274] Lustre: Skipped 1 previous similar message [ 7822.883196] Lustre: lustre-MDT0001: new disk, initializing [ 7822.912279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7822.923336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7822.929380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7824.287108] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7828.180376] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7830.113557] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7832.553096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7832.692399] Lustre: lustre-OST0000: new disk, initializing [ 7832.694768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7832.698559] Lustre: 327151:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7832.737471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7834.859215] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7838.657272] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7841.784031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7841.788401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7841.821373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7852.003196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7852.003803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7852.005801] LustreError: Skipped 1 previous similar message [ 7852.010246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7852.013486] Lustre: Skipped 1 previous similar message [ 7855.893816] Lustre: server umount lustre-OST0000 complete [ 7858.656933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7858.660202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7858.665181] Lustre: Skipped 1 previous similar message [ 7858.667584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7862.241107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7862.241580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7863.323525] Lustre: server umount lustre-MDT0000 complete [ 7864.605278] LustreError: 325178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786451792 with bad export cookie 7078535801470251847 [ 7864.608687] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7864.610239] LustreError: 325178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7864.748741] Lustre: server umount lustre-MDT0001 complete [ 7868.530480] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 7869.608083] Key type lgssc unregistered [ 7869.741434] LNet: 329064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7869.743773] LNetError: 329064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7869.750281] LNet: Removed LNI 192.168.201.138@tcp [ 7870.097753] Key type .llcrypt unregistered [ 7870.099643] Key type ._llcrypt unregistered [ 7878.261944] Key type ._llcrypt registered [ 7878.262935] Key type .llcrypt registered [ 7878.298490] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7878.622200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7878.649059] alg: No test for adler32 (adler32-zlib) [ 7879.507911] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 7879.597360] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 7881.176173] Key type lgssc registered [ 7881.553346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7884.281634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7884.288727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7885.437105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7886.756982] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7889.350118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7889.457667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7890.897307] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7893.196996] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7894.868345] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7897.000287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7897.099194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7898.833512] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7901.133046] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7915.056691] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7915.617904] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7917.112872] Lustre: server umount lustre-MDT0000 complete [ 7918.276064] LustreError: 330339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786451846 with bad export cookie 17471703717262933023 [ 7918.280066] LustreError: 330339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7918.280453] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7920.609326] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7920.609942] LustreError: 332800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7920.613420] Lustre: Skipped 1 previous similar message [ 7920.613981] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7920.618621] LustreError: 332800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7922.065233] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.38@tcp (stopping) [ 7922.070629] LustreError: 332800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7923.616709] LustreError: 330308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7923.617063] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7924.450743] Lustre: server umount lustre-MDT0001 complete [ 7927.189924] Lustre: lustre-OST0000: Not available for connect from 192.168.201.38@tcp (stopping) [ 7931.677264] Lustre: server umount lustre-OST0000 complete [ 7934.473589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7935.829054] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7936.781063] Lustre: 334509:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.138@tcp [ 7937.543065] Lustre: server umount MGS complete [ 7940.500219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7940.654350] LustreError: 335080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7940.677760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7942.105295] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7945.105019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7945.237910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7946.670476] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7949.560823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7951.651793] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7954.093597] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7968.068840] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 13 sec [ 7969.982540] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7970.504141] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7973.711093] Lustre: server umount lustre-OST0000 complete [ 7975.393839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7975.394616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7975.397898] Lustre: Skipped 1 previous similar message [ 7975.402342] Lustre: Skipped 1 previous similar message [ 7985.633103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7985.635580] Lustre: Skipped 3 previous similar messages [ 7985.661226] Lustre: server umount lustre-MDT0000 complete [ 7986.906025] LustreError: 335060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786451915 with bad export cookie 17471703717262933849 [ 7986.908161] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7986.909866] LustreError: 335060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7997.457435] Lustre: server umount lustre-MDT0001 complete [ 8001.207271] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8002.313966] Key type lgssc unregistered [ 8002.446398] LNet: 338538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8002.448962] LNetError: 338538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8002.459322] LNet: Removed LNI 192.168.201.138@tcp [ 8002.789505] Key type .llcrypt unregistered [ 8002.790935] Key type ._llcrypt unregistered [ 8009.049374] Key type ._llcrypt registered [ 8009.050371] Key type .llcrypt registered [ 8009.093646] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 8015.580877] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8015.932404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8015.984908] alg: No test for adler32 (adler32-zlib) [ 8016.847916] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 8016.947841] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 8018.528122] Key type lgssc registered [ 8018.889729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8031.069674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8031.075552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8032.166656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8032.177988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8032.219272] Lustre: lustre-MDT0000: new disk, initializing [ 8032.244385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8032.251630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8033.752760] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8038.051328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8038.099749] Lustre: 341671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8038.123548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8038.125112] Lustre: Skipped 1 previous similar message [ 8038.159390] Lustre: lustre-MDT0001: new disk, initializing [ 8038.193638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8038.207719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8038.212125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8039.590581] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8043.358173] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8045.271937] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8047.607323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8047.722584] Lustre: lustre-OST0000: new disk, initializing [ 8047.724149] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8047.727624] Lustre: 343639:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8047.754458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8049.644946] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8051.578885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8051.584169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8051.618226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8053.358152] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8054.807074] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8054.879463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8056.242953] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8056.312100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8061.922564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8061.923307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8061.924051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8061.925948] Lustre: Skipped 1 previous similar message [ 8061.927770] LustreError: Skipped 1 previous similar message [ 8063.054902] Lustre: server umount lustre-OST0000 complete [ 8067.041889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8067.042596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8067.043100] Lustre: Skipped 1 previous similar message [ 8067.053039] Lustre: Skipped 1 previous similar message [ 8070.433187] Lustre: server umount lustre-MDT0000 complete [ 8071.714324] LustreError: 342467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786451999 with bad export cookie 12380226387044604048 [ 8071.718488] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8071.718943] LustreError: 342467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8071.846524] Lustre: server umount lustre-MDT0001 complete [ 8074.858499] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 08:40:02 (1786452002) [ 8080.478083] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 8083.867857] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8096.149070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8096.228330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8096.236724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8096.277386] Lustre: lustre-MDT0000: new disk, initializing [ 8096.297041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8096.302073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8097.683582] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8101.842301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8101.886076] Lustre: 348387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8101.913383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8101.915380] Lustre: Skipped 1 previous similar message [ 8101.956031] Lustre: lustre-MDT0001: new disk, initializing [ 8101.992728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8102.006203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8102.011801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8103.382413] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8107.176231] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8109.121062] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8111.495585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8111.606873] Lustre: lustre-OST0000: new disk, initializing [ 8111.608289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8111.610527] Lustre: 350357:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8111.638168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8113.707061] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8117.426475] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8119.800668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8119.804515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8119.837045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8135.139051] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8135.139570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8135.142397] LustreError: Skipped 1 previous similar message [ 8135.148450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8135.152341] Lustre: Skipped 2 previous similar messages [ 8138.259508] Lustre: server umount lustre-OST0000 complete [ 8140.257827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8140.258732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8140.262435] Lustre: Skipped 2 previous similar messages [ 8145.377039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8145.379931] Lustre: Skipped 2 previous similar messages [ 8145.686467] Lustre: server umount lustre-MDT0000 complete [ 8146.944326] LustreError: 351225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786452075 with bad export cookie 12380226387044604937 [ 8146.946795] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8146.947787] LustreError: 351225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8147.082343] Lustre: server umount lustre-MDT0001 complete [ 8150.776171] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8151.790136] Key type lgssc unregistered [ 8151.913384] LNet: 352507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8151.915513] LNetError: 352507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8151.922264] LNet: Removed LNI 192.168.201.138@tcp [ 8152.231552] Key type .llcrypt unregistered [ 8152.232535] Key type ._llcrypt unregistered [ 8184.878419] Key type ._llcrypt registered [ 8184.879762] Key type .llcrypt registered [ 8184.923655] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8185.047435] Key type .llcrypt unregistered [ 8185.048672] Key type ._llcrypt unregistered [ 8212.947778] Key type ._llcrypt registered [ 8212.948946] Key type .llcrypt registered [ 8212.987585] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8213.103515] Key type .llcrypt unregistered [ 8213.104686] Key type ._llcrypt unregistered [ 8224.173870] Key type ._llcrypt registered [ 8224.174952] Key type .llcrypt registered [ 8224.216504] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8224.333200] Key type .llcrypt unregistered [ 8224.334614] Key type ._llcrypt unregistered [ 8235.264123] Key type ._llcrypt registered [ 8235.265124] Key type .llcrypt registered [ 8235.306878] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8235.433450] Key type .llcrypt unregistered [ 8235.435104] Key type ._llcrypt unregistered [ 8263.056540] Key type ._llcrypt registered [ 8263.057857] Key type .llcrypt registered [ 8263.101398] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8263.218368] Key type .llcrypt unregistered [ 8263.219408] Key type ._llcrypt unregistered [ 8274.689370] Key type ._llcrypt registered [ 8274.691055] Key type .llcrypt registered [ 8274.725873] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8274.829315] Key type .llcrypt unregistered [ 8274.830407] Key type ._llcrypt unregistered [ 8286.025358] Key type ._llcrypt registered [ 8286.026938] Key type .llcrypt registered [ 8286.064710] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8286.187369] Key type .llcrypt unregistered [ 8286.188576] Key type ._llcrypt unregistered [ 8297.136156] Key type ._llcrypt registered [ 8297.137214] Key type .llcrypt registered [ 8297.175545] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8297.286341] Key type .llcrypt unregistered [ 8297.287363] Key type ._llcrypt unregistered [ 8308.514789] Key type ._llcrypt registered [ 8308.515762] Key type .llcrypt registered [ 8308.561775] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8308.680325] Key type .llcrypt unregistered [ 8308.681595] Key type ._llcrypt unregistered [ 8319.523126] Key type ._llcrypt registered [ 8319.524913] Key type .llcrypt registered [ 8319.562499] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8319.676184] Key type .llcrypt unregistered [ 8319.677604] Key type ._llcrypt unregistered [ 8341.058328] Key type ._llcrypt registered [ 8341.059546] Key type .llcrypt registered [ 8341.101174] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8341.224219] Key type .llcrypt unregistered [ 8341.225210] Key type ._llcrypt unregistered [ 8352.644287] Key type ._llcrypt registered [ 8352.645345] Key type .llcrypt registered [ 8352.678578] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8352.773365] Key type .llcrypt unregistered [ 8352.774433] Key type ._llcrypt unregistered [ 8363.631402] Key type ._llcrypt registered [ 8363.632433] Key type .llcrypt registered [ 8363.667024] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8363.781343] Key type .llcrypt unregistered [ 8363.782416] Key type ._llcrypt unregistered [ 8382.244401] Key type ._llcrypt registered [ 8382.245888] Key type .llcrypt registered [ 8382.283681] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 8388.622628] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8389.117333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8389.158921] alg: No test for adler32 (adler32-zlib) [ 8390.024823] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 8390.138639] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 8391.728140] Key type lgssc registered [ 8392.075141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8403.568937] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8403.574542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8404.660569] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8404.670620] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8404.708621] Lustre: tst250fs-MDT0000: new disk, initializing [ 8404.729729] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8404.735808] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8406.121751] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8410.296188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8410.345137] Lustre: 372460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8410.369183] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8410.370819] Lustre: Skipped 1 previous similar message [ 8410.409324] Lustre: tst250fs-MDT0001: new disk, initializing [ 8410.446176] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8410.459492] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8410.463204] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8411.873862] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8414.128183] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8416.216239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8416.315841] Lustre: tst250fs-OST0000: new disk, initializing [ 8416.317399] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8416.319532] Lustre: 374092:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8416.347676] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8418.370143] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8421.878079] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8421.881115] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8421.918533] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8422.653247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8422.720114] Lustre: tst250fs-OST0001: new disk, initializing [ 8422.721916] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8422.724904] Lustre: 374961:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8422.762417] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8424.777836] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8427.332277] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8427.406458] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8427.410401] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8427.465702] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8437.731911] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.734918] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8437.738056] Lustre: Skipped 3 previous similar messages [ 8437.740808] Lustre: Skipped 2 previous similar messages [ 8442.850194] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8442.853096] Lustre: Skipped 4 previous similar messages [ 8443.561506] Lustre: server umount tst250fs-MDT0000 complete [ 8444.893335] LustreError: 373256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786452373 with bad export cookie 9529619270440609659 [ 8444.896852] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8444.900289] LustreError: 373256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8445.031134] Lustre: server umount tst250fs-MDT0001 complete [ 8456.196787] Lustre: server umount tst250fs-OST0000 complete [ 8467.691388] Lustre: server umount tst250fs-OST0001 complete [ 8473.675326] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 8476.538158] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8486.679636] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 08:46:54 (1786452414) [ 8487.197331] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8492.595595] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 8495.879199] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8508.084699] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8244 sec ========== 08:47:15 (1786452435) [ 8509.061590] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:47:16 (1786452436) === [ 8510.772863] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:47:18 (1786452438) === [ 8530.770137] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8532.481290] Key type lgssc unregistered [ 8532.645716] LNet: 384084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8532.649631] LNetError: 384084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8532.658100] LNet: Removed LNI 192.168.201.138@tcp [ 8533.103140] Key type .llcrypt unregistered [ 8533.104680] Key type ._llcrypt unregistered