[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479250165 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2624MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003211] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.007675] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008035] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010161] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013071] *** VALIDATE selinux *** [ 0.021145] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026605] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] *** VALIDATE tmpfs *** [ 0.031389] *** VALIDATE proc *** [ 0.032212] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.034235] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.040653] debug: unmapping init [mem 0xffffffff90059000-0xffffffff90060fff] [ 0.042187] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043725] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044022] ... version: 2 [ 0.045013] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047015] ... value mask: 0000ffffffffffff [ 0.048011] ... max period: 00007fffffffffff [ 0.049009] ... fixed-purpose events: 3 [ 0.049861] ... event mask: 000000070000000f [ 0.050308] rcu: Hierarchical SRCU implementation. [ 0.052494] smp: Bringing up secondary CPUs ... [ 0.053585] x86: Booting SMP configuration: [ 0.054026] .... node #0, CPUs: #1 #2 #3 [ 0.057301] smp: Brought up 1 node, 4 CPUs [ 0.059013] smpboot: Max logical packages: 1 [ 0.060019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141019] node 0 deferred pages initialised in 79ms [ 0.144358] devtmpfs: initialized [ 0.145224] x86/mm: Memory block size: 128MB [ 0.147734] gcov: version magic: 0x41383552 [ 0.149465] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151230] pinctrl core: initialized pinctrl subsystem [ 0.152140] [ 0.152547] ************************************************************* [ 0.153009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154010] ** ** [ 0.155009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156008] ** ** [ 0.157009] ** This means that this kernel is built to expose internal ** [ 0.158010] ** IOMMU data structures, which may compromise security on ** [ 0.159013] ** your system. ** [ 0.160012] ** ** [ 0.161009] ** If you see this message and you are not debugging the ** [ 0.162010] ** kernel, report this immediately to your vendor! ** [ 0.163009] ** ** [ 0.164007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165007] ************************************************************* [ 0.166566] NET: Registered protocol family 16 [ 0.167298] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168033] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.171087] cpuidle: using governor menu [ 0.172510] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.173310] PCI: Using configuration type 1 for base access [ 0.174093] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.180072] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.181019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.182169] cryptd: max_cpu_qlen set to 1000 [ 0.184153] ACPI: Added _OSI(Module Device) [ 0.185011] ACPI: Added _OSI(Processor Device) [ 0.186012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.187012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.190950] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.193096] ACPI: Interpreter enabled [ 0.194053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.195012] ACPI: Using IOAPIC for interrupt routing [ 0.196097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.197388] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.206379] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.207048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.208020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.209145] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.211147] acpiphp: Slot [2] registered [ 0.212087] acpiphp: Slot [5] registered [ 0.213080] acpiphp: Slot [6] registered [ 0.214091] acpiphp: Slot [7] registered [ 0.215021] acpiphp: Slot [8] registered [ 0.216134] acpiphp: Slot [9] registered [ 0.217162] acpiphp: Slot [10] registered [ 0.218092] acpiphp: Slot [3] registered [ 0.219162] acpiphp: Slot [4] registered [ 0.220089] acpiphp: Slot [11] registered [ 0.221092] acpiphp: Slot [12] registered [ 0.222135] acpiphp: Slot [13] registered [ 0.223102] acpiphp: Slot [14] registered [ 0.224100] acpiphp: Slot [15] registered [ 0.225086] acpiphp: Slot [16] registered [ 0.226097] acpiphp: Slot [17] registered [ 0.227102] acpiphp: Slot [18] registered [ 0.228114] acpiphp: Slot [19] registered [ 0.229092] acpiphp: Slot [20] registered [ 0.230089] acpiphp: Slot [21] registered [ 0.231110] acpiphp: Slot [22] registered [ 0.232094] acpiphp: Slot [23] registered [ 0.233097] acpiphp: Slot [24] registered [ 0.234087] acpiphp: Slot [25] registered [ 0.235119] acpiphp: Slot [26] registered [ 0.236094] acpiphp: Slot [27] registered [ 0.237096] acpiphp: Slot [28] registered [ 0.238091] acpiphp: Slot [29] registered [ 0.239102] acpiphp: Slot [30] registered [ 0.240096] acpiphp: Slot [31] registered [ 0.241072] PCI host bridge to bus 0000:00 [ 0.242020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.243092] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.244025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.245015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.246023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.247024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.248197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.250011] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.252192] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.258013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.261058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.262016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.263017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.264016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.265627] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.266929] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.267045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.268912] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.271014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.279927] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.281967] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.285662] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.288015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.291014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.298016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.303552] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.306022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.309015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.316020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.322319] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.325066] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.328027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.335017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.341872] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.349018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.355016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.373018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.393448] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.411014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.421014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.446019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.456073] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.462013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.467014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.477013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.484748] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.486377] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.489346] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.492343] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.494239] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.498066] iommu: Default domain type: Passthrough [ 0.499395] SCSI subsystem initialized [ 0.501076] ACPI: bus type USB registered [ 0.502086] usbcore: registered new interface driver usbfs [ 0.503069] usbcore: registered new interface driver hub [ 0.504052] usbcore: registered new device driver usb [ 0.505130] pps_core: LinuxPPS API ver. 1 registered [ 0.506005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.508056] PTP clock support registered [ 0.509102] EDAC MC: Ver: 3.0.0 [ 0.510299] PCI: Using ACPI for IRQ routing [ 0.511631] NetLabel: Initializing [ 0.513009] NetLabel: domain hash size = 128 [ 0.514006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.515057] NetLabel: unlabeled traffic allowed by default [ 0.516124] vgaarb: loaded [ 0.517234] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.518006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.525277] clocksource: Switched to clocksource kvm-clock [ 0.623485] VFS: Disk quotas dquot_6.6.0 [ 0.624990] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.627366] *** VALIDATE ramfs *** [ 0.628502] *** VALIDATE hugetlbfs *** [ 0.630020] pnp: PnP ACPI init [ 0.633041] pnp: PnP ACPI: found 6 devices [ 0.650321] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.653485] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.655538] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.657265] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.658636] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.660098] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.661692] NET: Registered protocol family 2 [ 0.663289] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.669417] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.674034] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.680346] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.683899] TCP: Hash tables configured (established 65536 bind 65536) [ 0.686657] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.689924] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.692643] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.695618] NET: Registered protocol family 1 [ 0.699193] RPC: Registered named UNIX socket transport module. [ 0.701121] RPC: Registered udp transport module. [ 0.702781] RPC: Registered tcp transport module. [ 0.704432] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.706853] NET: Registered protocol family 44 [ 0.708568] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.710830] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.712517] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.715028] PCI: CLS 0 bytes, default 64 [ 0.716718] Unpacking initramfs... [ 2.076126] debug: unmapping init [mem 0xffff9f45fcc54000-0xffff9f45fffbffff] [ 2.079976] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.082295] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.085307] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.560153] Initialise system trusted keyrings [ 2.561568] Key type blacklist registered [ 2.563290] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.571797] zbud: loaded [ 2.573941] *** VALIDATE nfs *** [ 2.574846] *** VALIDATE nfs4 *** [ 2.575984] pstore: using deflate compression [ 2.578582] Platform Keyring initialized [ 2.669340] NET: Registered protocol family 38 [ 2.671133] Key type asymmetric registered [ 2.672734] Asymmetric key parser 'x509' registered [ 2.674739] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.677909] io scheduler mq-deadline registered [ 2.679562] io scheduler kyber registered [ 2.681084] io scheduler bfq registered [ 2.683544] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.686358] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.688711] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.691128] ACPI: Power Button [PWRF] [ 2.696815] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.703139] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.719246] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.727824] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.740430] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.764464] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.791052] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.795512] Non-volatile memory driver v1.3 [ 2.796855] Linux agpgart interface v0.103 [ 2.824330] virtio_blk virtio1: [vda] 134472 512-byte logical blocks (68.8 MB/65.7 MiB) [ 2.827224] vda: detected capacity change from 0 to 68849664 [ 2.842887] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.845780] vdb: detected capacity change from 0 to 1073741824 [ 2.860702] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.864366] vdc: detected capacity change from 0 to 2621440000 [ 2.887411] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.889480] vdd: detected capacity change from 0 to 2621440000 [ 2.911414] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.914340] vde: detected capacity change from 0 to 4294967296 [ 2.927454] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.930327] vdf: detected capacity change from 0 to 4294967296 [ 2.938448] libphy: Fixed MDIO Bus: probed [ 2.946481] usbcore: registered new interface driver usbserial_generic [ 2.948490] usbserial: USB Serial support registered for generic [ 2.950394] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.954375] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.955891] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.957874] mousedev: PS/2 mouse device common for all mice [ 2.962269] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.965558] rtc_cmos 00:05: RTC can wake from S4 [ 2.969519] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.970161] rtc_cmos 00:05: registered as rtc0 [ 2.974154] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.974824] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.976980] intel_pstate: CPU model not supported [ 2.981949] hid: raw HID events driver (C) Jiri Kosina [ 2.983340] usbcore: registered new interface driver usbhid [ 2.984624] usbhid: USB HID core driver [ 2.985619] drop_monitor: Initializing network drop monitor service [ 2.986923] Initializing XFRM netlink socket [ 2.988135] NET: Registered protocol family 10 [ 2.990776] Segment Routing with IPv6 [ 2.992090] NET: Registered protocol family 17 [ 2.993925] mpls_gso: MPLS GSO support [ 2.999736] RAS: Correctable Errors collector initialized. [ 3.001801] AVX version of gcm_enc/dec engaged. [ 3.003637] AES CTR mode by8 optimization enabled [ 3.065932] sched_clock: Marking stable (3065915420, 0)->(4076580486, -1010665066) [ 3.068161] registered taskstats version 1 [ 3.069759] Loading compiled-in X.509 certificates [ 3.071090] zswap: loaded using pool lzo/zbud [ 3.091725] Key type big_key registered [ 3.102615] Key type encrypted registered [ 3.103672] ima: No TPM chip found, activating TPM-bypass! [ 3.104983] ima: Allocated hash algorithm: sha1 [ 3.105961] ima: No architecture policies found [ 3.107037] evm: Initialising EVM extended attributes: [ 3.108218] evm: security.selinux [ 3.108945] evm: security.ima [ 3.109653] evm: security.capability [ 3.110510] evm: HMAC attrs: 0x1 [ 3.112149] rtc_cmos 00:05: setting system clock to 2026-09-05 08:46:40 UTC (1788598000) [ 3.116154] debug: unmapping init [mem 0xffffffff91003000-0xffffffff911fffff] [ 3.118142] debug: unmapping init [mem 0xffffffff8fd82000-0xffffffff90058fff] [ 3.126082] Write protecting the kernel read-only data: 28672k [ 3.128452] debug: unmapping init [mem 0xffffffff8e403000-0xffffffff8e5fffff] [ 3.130270] debug: unmapping init [mem 0xffffffff8ed14000-0xffffffff8edfffff] [ 3.156383] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.161776] systemd[1]: Detected virtualization kvm. [ 3.163030] systemd[1]: Detected architecture x86-64. [ 3.164162] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.189358] systemd[1]: No hostname configured. [ 3.190462] systemd[1]: Set hostname to . [ 3.191738] random: systemd: uninitialized urandom read (16 bytes read) [ 3.193251] systemd[1]: Initializing machine ID from random generator. [ 3.224814] random: ln: uninitialized urandom read (6 bytes read) [ 3.299704] random: systemd: uninitialized urandom read (16 bytes read) [ 3.302528] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.308210] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.312105] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.801882] device-mapper: uevent: version 1.0.3 [ 3.803622] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.596955] virtio_net virtio0 ens2: renamed from eth0 [ 4.603211] random: fast init done [ 4.690893] scsi host0: ata_piix [ 4.810684] scsi host1: ata_piix [ 4.812219] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.814906] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.681181] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.618468] random: crng init done [ 9.629412] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.490263] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.216741] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.213123] SELinux: Disabled at runtime. [ 14.355230] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.386619] systemd[1]: Detected virtualization kvm. [ 14.395265] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.948479] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.951976] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.964769] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.969764] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.972785] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.980089] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.004334] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ 16.238625] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK [0[ 17.412509] squashfs: version 4.0 (2009/01/31) Phillip Lougher m] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.889597] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.173160] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.712470] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.820323] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 24.418929] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 24.956923] NFS: Registering the id_resolver key type [ 24.960256] Key type id_resolver registered [ 24.962765] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg134-server login: [ 91.436196] hrtimer: interrupt took 13403428 ns [ 98.437792] libcfs: loading out-of-tree module taints kernel. [ 98.473423] Key type ._llcrypt registered [ 98.475180] Key type .llcrypt registered [ 98.527823] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 118.901880] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 120.036719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 120.055839] alg: No test for adler32 (adler32-zlib) [ 121.654331] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 122.948475] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 124.911133] Key type lgssc registered [ 126.951093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.624426] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 185.962831] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 198.288948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 198.330802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.683972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 199.713487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 199.805574] Lustre: lustre-MDT0000: new disk, initializing [ 199.873631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.885884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 204.384635] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.192456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.275228] Lustre: 6513:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.300824] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.304679] Lustre: Skipped 1 previous similar message [ 216.371356] Lustre: lustre-MDT0001: new disk, initializing [ 216.415864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.432349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 216.438271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 219.779598] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.534790] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 231.452710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.692582] Lustre: lustre-OST0000: new disk, initializing [ 231.699629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 231.750596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.041438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 236.047443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 236.125067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 237.308772] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.546749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.701578] Lustre: lustre-OST0001: new disk, initializing [ 250.707159] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 250.767802] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.276813] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.161464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 260.171396] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 260.232749] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 268.421156] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 274.073471] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 285.616996] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing check_logdir /tmp/testlogs/ [ 291.062548] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing yml_node [ 296.862275] Lustre: DEBUG MARKER: Client: 2.17.51.74 [ 299.533768] Lustre: DEBUG MARKER: MDS: 2.17.51.74 [ 302.461302] Lustre: DEBUG MARKER: OSS: 2.17.51.74 [ 303.849667] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 04:51:40 EDT 2026 [ 323.050295] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 324.934266] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 344.548019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 344.559652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.585429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.108704] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.109944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.131201] Lustre: Skipped 1 previous similar message [ 347.149492] Lustre: Skipped 3 previous similar messages [ 349.207905] Lustre: server umount lustre-MDT0000 complete [ 354.449879] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598351 with bad export cookie 4343970695951807251 [ 354.461100] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 354.469407] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 357.343808] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.347334] LustreError: 10446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.354354] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 357.373143] Lustre: Skipped 2 previous similar messages [ 357.432328] LustreError: 10446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 359.392629] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 359.396791] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 359.414788] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 359.431428] Lustre: Skipped 2 previous similar messages [ 361.177371] Lustre: server umount lustre-MDT0001 complete [ 366.472786] Lustre: server umount lustre-OST0000 complete [ 371.053180] Lustre: server umount lustre-OST0001 complete [ 377.579858] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 390.097511] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 435.994245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 436.512607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 436.565937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 436.657534] Lustre: lustre-MDT0000: new disk, initializing [ 436.726906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 436.742948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 441.939765] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 452.002701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.128207] Lustre: 14992:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 452.233835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 452.244825] Lustre: Skipped 1 previous similar message [ 452.382831] Lustre: lustre-MDT0001: new disk, initializing [ 452.565033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 452.646283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 452.663158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 457.693718] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 465.705527] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 470.131343] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 477.067502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 477.377611] Lustre: lustre-OST0000: new disk, initializing [ 477.383415] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 477.469551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.461512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 479.472933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 479.647834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 483.015211] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 491.581815] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 496.532545] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 496.815139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 501.575594] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 501.785576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 505.313914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 505.321161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.337852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 505.339396] Lustre: Skipped 1 previous similar message [ 505.362966] Lustre: Skipped 1 previous similar message [ 515.563589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 515.567547] Lustre: Skipped 3 previous similar messages [ 518.112933] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 518.319096] Lustre: server umount lustre-OST0000 complete [ 524.280269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 524.291285] LustreError: Skipped 1 previous similar message [ 524.301384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.898694] Lustre: server umount lustre-MDT0000 complete [ 530.914324] LustreError: 15000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.947323] LustreError: 15000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 532.752063] LustreError: 14984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598530 with bad export cookie 4343970695951808784 [ 532.765448] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 532.772486] LustreError: 14984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 533.283356] Lustre: server umount lustre-MDT0001 complete [ 539.937760] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:55:36 (1788598536) [ 559.456909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.843823] LustreError: 19465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.912468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 564.547687] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 565.220727] LustreError: 19466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.134919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.469988] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 584.032178] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 589.093333] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 592.217061] Lustre: Failing over lustre-MDT0000 [ 592.703467] Lustre: server umount lustre-MDT0000 complete [ 593.900063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 593.909513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.934832] Lustre: Skipped 1 previous similar message [ 593.952910] LustreError: 20176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.969559] LustreError: 20176:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 611.295505] Lustre: 3668:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788598592/real 1788598592] req@ffff9f45422a0e00 x1875481062437888/t0(0) o400->MGC192.168.201.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788598608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 611.320746] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.343662] LustreError: 19462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.361698] LustreError: 19462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 612.319885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.814437] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 626.251850] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 627.169958] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 627.172928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 627.212703] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 634.440540] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 638.757951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.766953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 638.772455] Lustre: Skipped 2 previous similar messages [ 638.777396] Lustre: Skipped 3 previous similar messages [ 653.281266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 653.503578] Lustre: server umount lustre-MDT0000 complete [ 657.296794] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598654 with bad export cookie 4343970695951810198 [ 657.312478] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 657.316494] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 657.683743] Lustre: server umount lustre-MDT0001 complete [ 666.849644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.350379] LustreError: 22466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 667.363117] LustreError: 22466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 667.465655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 667.474366] Lustre: Skipped 2 previous similar messages [ 671.332487] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 680.872372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.906636] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 694.799242] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 701.139769] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 704.018290] Lustre: Failing over lustre-MDT0000 [ 704.443351] Lustre: server umount lustre-MDT0000 complete [ 706.083627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.097078] Lustre: Skipped 1 previous similar message [ 722.271664] Lustre: 3665:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788598703/real 1788598703] req@ffff9f467ad38e00 x1875481062467712/t0(0) o400->MGC192.168.201.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788598719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 722.338726] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 724.302362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.619632] LustreError: 22461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 731.619937] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3c48e25526ea0240 [ 731.653718] LustreError: 22461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 731.693107] Lustre: MGC192.168.201.134@tcp: Connection restored to 0@lo (at 0@lo) [ 731.716443] Lustre: Skipped 1 previous similar message [ 732.238946] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 737.263690] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 737.266602] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 737.289858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 737.318724] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 746.381451] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 752.609988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 752.625175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 752.627745] Lustre: Skipped 7 previous similar messages [ 757.474690] Lustre: server umount lustre-MDT0000 complete [ 761.326459] LustreError: 23185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598758 with bad export cookie 4343970695951811136 [ 761.329156] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.339768] LustreError: 23185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 761.715287] Lustre: server umount lustre-MDT0001 complete [ 770.035181] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:59:26 (1788598766) [ 786.854473] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 799.020799] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 847.265654] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 860.199051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.455200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 860.559701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 860.665035] Lustre: lustre-MDT0000: new disk, initializing [ 860.810431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 860.817769] Lustre: Skipped 2 previous similar messages [ 860.844776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 866.112894] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 877.274815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.387045] Lustre: 28475:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 877.755730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 881.810729] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 891.662160] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 896.936584] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 904.641521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 906.318565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 906.365223] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 906.488874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 906.501699] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 910.698133] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 919.117340] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 925.011357] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 947.685034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 947.701283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.706625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 947.717851] Lustre: Skipped 3 previous similar messages [ 947.730930] Lustre: Skipped 2 previous similar messages [ 952.775199] Lustre: server umount lustre-OST0000 complete [ 964.961945] Lustre: server umount lustre-MDT0000 complete [ 968.162225] LustreError: 28486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.194529] LustreError: 28486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 969.697688] LustreError: 29264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598967 with bad export cookie 4343970695951811549 [ 969.715554] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 969.716177] LustreError: 29264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 970.169994] Lustre: server umount lustre-MDT0001 complete [ 985.554502] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 988.626470] Key type lgssc unregistered [ 988.976280] LNet: 32078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.993834] LNetError: 32078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.033273] LNet: Removed LNI 192.168.201.134@tcp [ 990.493868] Key type .llcrypt unregistered [ 990.500096] Key type ._llcrypt unregistered [ 1020.277538] Key type ._llcrypt registered [ 1020.280218] Key type .llcrypt registered [ 1020.420066] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1021.509840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1021.558865] alg: No test for adler32 (adler32-zlib) [ 1022.852994] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 1023.110563] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1024.751453] Key type lgssc registered [ 1026.071653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1034.929815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1034.968087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.610545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.071985] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1045.986697] LustreError: 33474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.998277] LustreError: 33474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1050.153169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1050.627361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1054.854403] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1060.990634] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1065.982180] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1073.759137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1074.156672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.227397] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1081.162753] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1081.430028] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1088.237485] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1105.053663] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 05:05:01 (1788599101) [ 1107.348935] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1109.083373] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1116.586412] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 05:05:13 (1788599113) [ 1126.996354] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 05:05:22 (1788599122) [ 1128.331757] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1139.670743] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1139.682689] Lustre: Skipped 2 previous similar messages [ 1142.381120] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 05:05:38 (1788599138) [ 1144.963270] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1144.968612] Lustre: Skipped 2 previous similar messages [ 1153.233207] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1153.236874] Lustre: Skipped 1 previous similar message [ 1169.764822] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1169.769690] Lustre: Skipped 5 previous similar messages [ 1174.104777] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 05:06:09 (1788599169) [ 1177.992577] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1181.824848] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1193.424349] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 05:06:29 (1788599189) [ 1207.629759] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1207.637210] Lustre: Skipped 7 previous similar messages [ 1209.675938] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 05:06:45 (1788599205) [ 1228.773309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1228.781284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.796659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1229.801340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1229.805922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1229.819820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.844072] Lustre: Skipped 1 previous similar message [ 1231.870390] Lustre: server umount lustre-OST0000 complete [ 1240.033343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1240.035432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.066089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1240.079454] Lustre: Skipped 1 previous similar message [ 1240.099328] Lustre: Skipped 1 previous similar message [ 1245.154306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1245.159367] Lustre: Skipped 1 previous similar message [ 1250.274071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1250.276270] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 1250.286029] Lustre: Skipped 1 previous similar message [ 1250.471432] Lustre: server umount lustre-MDT0000 complete [ 1254.496341] LustreError: 37057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599251 with bad export cookie 13897111830229477011 [ 1254.500559] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1254.510366] LustreError: 37057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1255.081353] Lustre: server umount lustre-MDT0001 complete [ 1266.615972] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 1268.967907] Key type lgssc unregistered [ 1269.282979] LNet: 40984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.303451] LNetError: 40984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.318151] LNet: Removed LNI 192.168.201.134@tcp [ 1270.028744] Key type .llcrypt unregistered [ 1270.029879] Key type ._llcrypt unregistered [ 1290.001785] Key type ._llcrypt registered [ 1290.003548] Key type .llcrypt registered [ 1290.105281] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 1306.075889] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1307.397568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1307.466749] alg: No test for adler32 (adler32-zlib) [ 1308.479167] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 1308.696775] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1310.391271] Key type lgssc registered [ 1311.918610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1353.411627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1353.428887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.640982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1354.678662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1354.762140] Lustre: lustre-MDT0000: new disk, initializing [ 1354.846984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.869676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1361.861374] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1375.858628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.067796] Lustre: 44117:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1376.178644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1376.184357] Lustre: Skipped 1 previous similar message [ 1376.339833] Lustre: lustre-MDT0001: new disk, initializing [ 1376.456227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1376.524959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1376.536718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1381.983551] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1391.046344] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1395.608499] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1403.717871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1404.029891] Lustre: lustre-OST0000: new disk, initializing [ 1404.034802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1404.120045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1410.116208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1410.118890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1410.205071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1411.414604] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1419.917658] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1424.244681] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1424.509851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1428.649460] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1428.896210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1431.014161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1431.014859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1431.022208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1431.032332] Lustre: Skipped 1 previous similar message [ 1436.128652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1436.130591] Lustre: Skipped 1 previous similar message [ 1441.257297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1441.266499] Lustre: Skipped 1 previous similar message [ 1445.343351] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1445.670855] Lustre: server umount lustre-OST0000 complete [ 1451.497820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.514748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1451.519335] Lustre: Skipped 1 previous similar message [ 1451.534461] Lustre: Skipped 1 previous similar message [ 1456.609971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.620280] Lustre: Skipped 1 previous similar message [ 1463.265981] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1463.415755] Lustre: server umount lustre-MDT0000 complete [ 1466.636628] LustreError: 44107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599464 with bad export cookie 17461495841099618836 [ 1466.648611] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1466.650379] LustreError: 44107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1466.953975] Lustre: server umount lustre-MDT0001 complete [ 1477.052617] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 05:11:12 (1788599472) [ 1484.765321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1485.438914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1489.379171] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1490.914686] LustreError: 48172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.949090] LustreError: 48172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1495.086893] LustreError: 48173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.701134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1499.066849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1504.080535] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1510.708696] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1515.474983] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1523.542108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1523.663674] LustreError: 49769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1523.944841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1530.489865] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1536.778257] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1541.195738] LustreError: 50366:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1542.012152] Lustre: Modifying parameter general.timeout=1 in log params [ 1543.729905] Lustre: Modifying parameter general.timeout=3 in log params [ 1543.734344] Lustre: Skipped 1 previous similar message [ 1546.436333] Lustre: Modifying parameter general.timeout=6 in log params [ 1546.439456] Lustre: Skipped 2 previous similar messages [ 1551.099632] Lustre: Modifying parameter general.timeout=12 in log params [ 1551.101742] Lustre: Skipped 5 previous similar messages [ 1559.371249] Lustre: Modifying parameter general.timeout=22 in log params [ 1559.376414] Lustre: Skipped 9 previous similar messages [ 1575.484371] Lustre: Modifying parameter general.timeout=43 in log params [ 1575.491507] Lustre: Skipped 20 previous similar messages [ 1608.023080] Lustre: Modifying parameter general.timeout=73 in log params [ 1608.025854] Lustre: Skipped 29 previous similar messages [ 1675.457972] Lustre: Disabling parameter general.timeout= in log params [ 1675.465032] Lustre: Skipped 56 previous similar messages [ 1678.594070] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 05:14:35 (1788599675) [ 1681.376123] Lustre: 57311:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1683.431321] Lustre: 57419:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1686.696271] Lustre: 57616:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1686.715458] Lustre: 57616:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1718.326841] Lustre: 59188:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1718.334613] Lustre: 59188:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 1718.355411] LustreError: 59188:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1722.324071] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 05:15:18 (1788599718) [ 1735.651537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1735.669790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1735.677194] Lustre: Skipped 2 previous similar messages [ 1735.691869] Lustre: Skipped 4 previous similar messages [ 1741.712411] Lustre: server umount lustre-MDT0000 complete [ 1745.894218] LustreError: 48172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1745.924795] LustreError: 48172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1745.956425] LustreError: 50415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599743 with bad export cookie 17461495841099619718 [ 1745.958261] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1745.969154] LustreError: 50415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1746.336375] Lustre: server umount lustre-MDT0001 complete [ 1761.573264] Lustre: server umount lustre-OST0000 complete [ 1791.788468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1792.174991] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1792.505030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.721878] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1806.655350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1806.718629] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1806.743797] Lustre: 61721:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1811.820690] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1818.441775] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1822.755588] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1829.664164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1829.842243] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1830.075101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.077863] Lustre: Skipped 1 previous similar message [ 1836.623536] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1843.178269] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1845.027566] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1845.036642] Lustre: Skipped 23 previous similar messages [ 1845.086107] Lustre: 63948:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1852.384837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1852.400679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1852.418257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1852.425849] Lustre: Skipped 3 previous similar messages [ 1857.889756] Lustre: server umount lustre-OST0000 complete [ 1863.137280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.137316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1863.152913] Lustre: Skipped 1 previous similar message [ 1863.157825] LustreError: Skipped 1 previous similar message [ 1867.918784] Lustre: server umount lustre-MDT0000 complete [ 1868.256543] LustreError: 61733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1868.269768] LustreError: 61733:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1871.461898] LustreError: 61713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599868 with bad export cookie 17461495841099622763 [ 1871.463937] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1871.476245] LustreError: 61713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1871.772449] Lustre: server umount lustre-MDT0001 complete [ 1884.351454] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 1887.810715] Key type lgssc unregistered [ 1888.158815] LNet: 65143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1888.165406] LNetError: 65143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1888.200150] LNet: Removed LNI 192.168.201.134@tcp [ 1889.048730] Key type .llcrypt unregistered [ 1889.058050] Key type ._llcrypt unregistered [ 1913.797368] Key type ._llcrypt registered [ 1913.801571] Key type .llcrypt registered [ 1913.892359] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1915.433516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1915.472296] alg: No test for adler32 (adler32-zlib) [ 1916.573183] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 1916.843541] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1918.551335] Key type lgssc registered [ 1919.597841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1926.662785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1926.677652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.073432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1931.884938] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1938.401952] LustreError: 66542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1938.411554] LustreError: 66542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1939.326236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1940.034400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1944.182973] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1950.021715] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1954.442706] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1961.697432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1962.183804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.702815] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1974.796623] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1982.875140] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1991.146724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1991.162323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1991.173896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1994.208558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1994.224764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.235706] Lustre: Skipped 1 previous similar message [ 1997.205328] Lustre: server umount lustre-MDT0000 complete [ 1999.334370] LustreError: 67548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1999.353270] LustreError: 67548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2000.995855] LustreError: 66522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599998 with bad export cookie 16919835073711374367 [ 2001.003431] LustreError: 66522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2001.010915] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2001.335881] Lustre: server umount lustre-MDT0001 complete [ 2015.029403] Lustre: server umount lustre-OST0000 complete [ 2041.036919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.308726] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2041.331424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2041.573742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2046.191952] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2055.253393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2055.314425] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2055.339515] Lustre: 71082:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2055.832832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2061.159752] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2066.792635] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2071.225512] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2079.090557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2079.238453] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2079.600308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2085.725238] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2091.908359] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2093.607898] Lustre: 73318:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2095.533720] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2102.759578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.764127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2102.764741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2102.764746] Lustre: Skipped 2 previous similar messages [ 2102.773923] Lustre: Skipped 1 previous similar message [ 2102.802594] LustreError: Skipped 1 previous similar message [ 2107.071391] Lustre: server umount lustre-OST0000 complete [ 2111.978624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2111.995197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.017396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2112.020064] Lustre: Skipped 1 previous similar message [ 2117.551485] Lustre: server umount lustre-MDT0000 complete [ 2118.113775] LustreError: 71094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.286271] LustreError: 71074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600118 with bad export cookie 16919835073711375207 [ 2121.293802] LustreError: 71074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2121.294795] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2121.590343] Lustre: server umount lustre-MDT0001 complete [ 2135.323362] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 2138.406408] Key type lgssc unregistered [ 2138.678756] LNet: 74514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.684069] LNetError: 74514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.700223] LNet: Removed LNI 192.168.201.134@tcp [ 2139.453542] Key type .llcrypt unregistered [ 2139.454724] Key type ._llcrypt unregistered [ 2164.095370] Key type ._llcrypt registered [ 2164.097446] Key type .llcrypt registered [ 2164.199403] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2165.057808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2165.089078] alg: No test for adler32 (adler32-zlib) [ 2166.279772] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 2166.433343] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 2168.056514] Key type lgssc registered [ 2169.142571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2182.112533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2182.173573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2184.186362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.734316] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2194.414739] LustreError: 75912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.442215] LustreError: 75912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2198.924700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.564959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2204.514946] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2212.652994] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2216.726522] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2224.099800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2224.635192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.029577] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2236.712741] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2252.179208] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2253.286965] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2255.192771] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2255.197424] Lustre: Skipped 1 previous similar message [ 2258.326729] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2258.331110] Lustre: Skipped 2 previous similar messages [ 2262.329960] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2262.334895] Lustre: Skipped 3 previous similar messages [ 2270.534349] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2270.540170] Lustre: Skipped 7 previous similar messages [ 2287.136547] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2287.142948] Lustre: Skipped 16 previous similar messages [ 2319.895977] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2319.903046] Lustre: Skipped 33 previous similar messages [ 2359.722168] Lustre: 83702:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2362.690141] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:25:58 (1788600358) [ 2364.609210] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2366.542739] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:26:02 (1788600362) [ 2372.076718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2372.077569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.095561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2372.099290] Lustre: Skipped 1 previous similar message [ 2372.109143] Lustre: Skipped 1 previous similar message [ 2376.766217] Lustre: server umount lustre-OST0000 complete [ 2382.314577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.325113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2382.331745] Lustre: Skipped 1 previous similar message [ 2387.424336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2392.545627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2392.551539] Lustre: Skipped 2 previous similar messages [ 2395.103211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2395.315728] Lustre: server umount lustre-MDT0000 complete [ 2397.664691] LustreError: 75913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.680617] LustreError: 75913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2399.299514] LustreError: 75893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600396 with bad export cookie 14516062963232533241 [ 2399.310589] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2399.632552] Lustre: server umount lustre-MDT0001 complete [ 2410.246105] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 2413.101185] Key type lgssc unregistered [ 2413.429604] LNet: 84936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2413.436614] LNetError: 84936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2413.457202] LNet: Removed LNI 192.168.201.134@tcp [ 2414.487206] Key type .llcrypt unregistered [ 2414.490057] Key type ._llcrypt unregistered [ 2418.571865] Key type ._llcrypt registered [ 2418.574287] Key type .llcrypt registered [ 2418.692525] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2422.270995] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules [ 2423.375364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2423.444036] alg: No test for adler32 (adler32-zlib) [ 2424.680859] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 2425.174284] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 2426.887419] Key type lgssc registered [ 2428.116538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2429.347884] LustreError: 86104:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 2449.556778] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2452.599611] LustreError: 86104:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2459.952881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2459.969301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.717081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.769418] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2470.880133] LustreError: 87090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.912752] LustreError: 87090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2476.006612] LustreError: 87091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.477580] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:27:53 (1788600473) [ 2479.010069] LustreError: 87090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.191371] LustreError: 87122:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2482.214927] LustreError: 87122:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2482.237111] LustreError: 87122:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2482.859032] Lustre: server umount lustre-MDT0000 complete [ 2497.524244] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 2500.636170] Key type lgssc unregistered [ 2500.993456] LNet: 88585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2501.001252] LNetError: 88585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2502.054276] LNet: Removed LNI 192.168.201.134@tcp [ 2502.964246] Key type .llcrypt unregistered [ 2502.977603] Key type ._llcrypt unregistered [ 2523.563807] Key type ._llcrypt registered [ 2523.567374] Key type .llcrypt registered [ 2523.700855] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2524.655979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2524.774148] alg: No test for adler32 (adler32-zlib) [ 2525.832517] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 2526.047685] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 2527.743257] Key type lgssc registered [ 2528.848358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2537.013728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2537.030488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.592615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.538263] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2548.199052] LustreError: 89791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.229092] LustreError: 89791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2552.100236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.747660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2556.828299] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2562.519333] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2566.623471] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2573.191531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2573.736206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2580.044924] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2585.666774] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2637.692722] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:30:33 (1788600633) [ 2640.413408] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2643.075203] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:30:38 (1788600638) [ 2649.575982] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2649.583859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2649.584195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2649.602833] Lustre: Skipped 2 previous similar messages [ 2649.635186] Lustre: Skipped 2 previous similar messages [ 2654.475364] Lustre: server umount lustre-MDT0000 complete [ 2654.691694] LustreError: 89788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2654.724510] LustreError: 89788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2657.978181] LustreError: 89771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600655 with bad export cookie 10923322294668735982 [ 2657.984069] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2657.989151] LustreError: 89771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2658.597820] Lustre: server umount lustre-MDT0001 complete [ 2673.345909] Lustre: server umount lustre-OST0000 complete [ 2686.005161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.702501] LustreError: 93750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.958340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2691.666416] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2692.072924] LustreError: 93751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2697.189092] LustreError: 93750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.640130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2701.315547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2706.695746] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2714.020408] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2718.298635] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2731.092508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2731.260722] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2731.272426] LustreError: 93737:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2731.310430] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2731.323466] LustreError: 95608:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 2731.329208] LustreError: 95608:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 2731.341268] LustreError: 95608:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 2731.442225] Lustre: server umount lustre-OST0000 complete [ 2731.449418] LustreError: 95608:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2739.142685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2739.356447] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2739.361743] LustreError: 93737:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2739.374617] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2739.390500] LustreError: 96069:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 2739.396442] LustreError: 96069:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 2739.401072] LustreError: 96069:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 2739.463870] Lustre: server umount lustre-OST0000 complete [ 2739.470059] LustreError: 96069:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2748.292102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2748.471064] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2748.474737] Lustre: Found index 0 for lustre-OST0000, updating log [ 2748.479573] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2748.546615] Lustre: lustre-OST0000: new disk, initializing [ 2748.550403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2748.762228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2754.853637] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2758.432286] Lustre: Failing over lustre-OST0000 [ 2758.639500] Lustre: server umount lustre-OST0000 complete [ 2762.208263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.211555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.217033] Lustre: Skipped 1 previous similar message [ 2762.222288] Lustre: Skipped 1 previous similar message [ 2767.329308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2767.341379] Lustre: Skipped 1 previous similar message [ 2771.424707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2771.439902] Lustre: Skipped 1 previous similar message [ 2772.628981] Lustre: server umount lustre-MDT0000 complete [ 2776.414917] LustreError: 93730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600773 with bad export cookie 10923322294668738726 [ 2776.434867] LustreError: 93730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2776.436700] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2776.826488] Lustre: server umount lustre-MDT0001 complete [ 2787.471295] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:33:03 (1788600783) [ 2804.542787] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2817.036884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.388289] LustreError: 98922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.509822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2822.768544] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2833.882211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2839.061863] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2842.330503] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2850.417937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2851.183035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2851.191929] Lustre: Skipped 1 previous similar message [ 2851.208295] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2858.312114] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2859.361424] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2859.392963] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2859.433982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2862.523346] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:06 [ 2862.547574] Lustre: lustre-OST0000: Recovery over after 0:03, of 1 clients 1 recovered and 0 were evicted. [ 2862.547800] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2862.559632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2862.568565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2871.533674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2871.769610] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2872.052935] Lustre: lustre-OST0001: new disk, initializing [ 2872.072216] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2880.480844] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2881.659711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2881.677256] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2881.814719] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2889.244994] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2898.041994] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2907.620119] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2907.634616] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2907.643623] Lustre: Skipped 1 previous similar message [ 2914.079569] Lustre: server umount lustre-MDT0001 complete [ 2917.870161] LustreError: 98923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2917.898126] LustreError: 98923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2925.463293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.601572] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2925.604085] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2925.711483] Lustre: 98910:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2926.541653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2926.550355] Lustre: Skipped 1 previous similar message [ 2927.762528] LustreError: 102760:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2927.771361] LustreError: 102760:0:(tgt_mount.c:666:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2927.785707] LustreError: 98914:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2927.916495] LustreError: 102769:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.201.134@tcp: cfg command failed: rc = -17 [ 2927.921651] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2927.921651] [ 2931.738383] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2932.274233] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2945.356030] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 05:35:41 (1788600941) [ 2965.976515] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2987.321734] Lustre: 104767:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3013.308602] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3017.683403] Lustre: 105902:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3115.510826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3115.516335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3115.531303] Lustre: Skipped 2 previous similar messages [ 3115.538322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3115.545757] Lustre: Skipped 5 previous similar messages [ 3118.298875] Lustre: server umount lustre-MDT0000 complete [ 3121.148274] LustreError: 98919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3121.174399] LustreError: 98919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3122.673576] LustreError: 100408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601120 with bad export cookie 10923322294668739650 [ 3122.675081] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3122.681101] LustreError: 100408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3123.029470] Lustre: server umount lustre-MDT0001 complete [ 3137.517611] Lustre: server umount lustre-OST0000 complete [ 3152.318334] Lustre: server umount lustre-OST0001 complete [ 3157.039569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3168.462672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3179.919501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3192.726458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3219.331180] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 3232.207341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.237863] Lustre: lustre-MDT0000: reset Object Index mappings [ 3232.785718] LustreError: 109847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.924107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3237.346357] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3247.972797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3247.995647] Lustre: lustre-MDT0001: reset Object Index mappings [ 3254.514437] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3258.357547] Lustre: 110962:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3266.252910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3272.630328] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3272.879230] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3278.023085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3283.853280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3285.910714] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3285.930056] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3292.561387] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3300.811552] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3304.199580] Lustre: 112821:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3360.250483] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:42:36 (1788601356) [ 3386.850564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3386.856159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.866570] Lustre: Skipped 3 previous similar messages [ 3386.873575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3386.877391] Lustre: Skipped 3 previous similar messages [ 3393.217082] Lustre: server umount lustre-MDT0000 complete [ 3397.433041] LustreError: 109828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601394 with bad export cookie 10923322294668786760 [ 3397.445850] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3397.446798] LustreError: 109828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3397.601065] LustreError: 112092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3397.605325] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.614931] LustreError: 112092:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3397.622619] Lustre: Skipped 4 previous similar messages [ 3404.294417] Lustre: server umount lustre-MDT0001 complete [ 3414.682774] Lustre: server umount lustre-OST0000 complete [ 3424.732991] Lustre: server umount lustre-OST0001 complete [ 3432.029093] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 3441.497449] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 3482.720919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.094673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3483.134093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3483.215895] Lustre: lustre-MDT0000: new disk, initializing [ 3483.306348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3489.202071] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3499.373282] Lustre: server umount lustre-MDT0000 complete [ 3510.510898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.103836] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3511.142850] Lustre: 117040:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3511.169499] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3511.548431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3511.551609] Lustre: Skipped 4 previous similar messages [ 3517.170339] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3522.946244] Lustre: server umount lustre-MDT0000 complete [ 3533.226782] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:45:29 (1788601529) [ 3535.016958] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3537.045803] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:45:33 (1788601533) [ 3557.305888] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 3569.075611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.276715] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3582.924542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3583.049522] Lustre: 119056:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3583.131446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3583.137502] Lustre: Skipped 1 previous similar message [ 3583.266056] Lustre: lustre-MDT0001: new disk, initializing [ 3583.457529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3583.470578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3587.715730] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3592.904544] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3600.227909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3600.501643] Lustre: lustre-OST0000: new disk, initializing [ 3600.506772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3602.495867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3602.500940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3602.612362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3605.860169] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3616.348768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3616.689603] Lustre: lustre-OST0001: new disk, initializing [ 3616.693209] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3617.960533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3617.975336] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3618.126151] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3624.219169] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3635.349501] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3639.580900] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4033.368518] Lustre: 127858:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4033.378930] Lustre: 127858:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4033.389793] Lustre: 127858:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4033.402386] Lustre: 127858:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4033.415684] Lustre: 127858:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4033.432403] Lustre: 127858:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4363.969640] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:59:19 (1788602359) [ 4382.178196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4382.189670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4382.197115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4382.204762] Lustre: Skipped 12 previous similar messages [ 4384.226508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4387.963159] Lustre: server umount lustre-MDT0000 complete [ 4389.362699] LustreError: 122244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.406684] LustreError: 122244:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4393.241596] LustreError: 119050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602390 with bad export cookie 10923322294668854149 [ 4393.250065] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4393.256382] LustreError: 119050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4394.467228] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4394.492743] Lustre: Skipped 3 previous similar messages [ 4394.501166] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4394.504853] Lustre: Skipped 4 previous similar messages [ 4400.510312] Lustre: server umount lustre-MDT0001 complete [ 4412.875191] Lustre: server umount lustre-OST0000 complete [ 4431.327146] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4431.499091] Lustre: server umount lustre-OST0001 complete [ 4440.130587] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 4456.299874] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4500.575140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.922687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4500.953691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4501.041863] Lustre: lustre-MDT0000: new disk, initializing [ 4501.154879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4501.163456] Lustre: Skipped 4 previous similar messages [ 4501.184801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4506.331258] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4517.836174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4517.979578] Lustre: 130922:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4518.026438] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4518.029830] Lustre: Skipped 1 previous similar message [ 4518.114868] Lustre: lustre-MDT0001: new disk, initializing [ 4518.308828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4518.318405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4523.187368] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4531.562900] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4536.020842] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4543.756708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4544.187449] Lustre: lustre-OST0000: new disk, initializing [ 4544.192171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4545.764517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4545.776769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4545.910103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4551.222369] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4561.014417] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4573.427868] Lustre: lustre-MDD0000: changelog on [ 4576.170323] Lustre: lustre-MDD0001: changelog on [ 6576.006530] Lustre: lustre-MDD0001: changelog off [ 6578.392955] Lustre: 134007:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6579.536694] Lustre: lustre-MDD0000: changelog off [ 6588.896548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6588.904451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6588.908733] LustreError: Skipped 1 previous similar message [ 6588.915410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6588.915422] Lustre: Skipped 2 previous similar messages [ 6588.944371] Lustre: Skipped 1 previous similar message [ 6592.411608] Lustre: server umount lustre-OST0000 complete [ 6597.088647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6597.092143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6597.121384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6597.129089] Lustre: Skipped 3 previous similar messages [ 6599.649549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6599.654430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6599.660785] Lustre: Skipped 1 previous similar message [ 6602.758668] Lustre: server umount lustre-MDT0000 complete [ 6604.769509] LustreError: 131780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6604.785893] LustreError: 131780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6606.645504] LustreError: 130914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604604 with bad export cookie 10923322294669678399 [ 6606.654862] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6606.660221] LustreError: 130914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6607.013722] Lustre: server umount lustre-MDT0001 complete [ 6621.136501] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 6624.628043] Key type lgssc unregistered [ 6624.911968] LNet: 141682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6624.917942] LNetError: 141682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6624.955254] LNet: Removed LNI 192.168.201.134@tcp [ 6626.021179] Key type .llcrypt unregistered [ 6626.024259] Key type ._llcrypt unregistered [ 6652.147983] Key type ._llcrypt registered [ 6652.159086] Key type .llcrypt registered [ 6652.428370] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 6673.555769] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 6676.351839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6676.536668] alg: No test for adler32 (adler32-zlib) [ 6678.088391] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 6678.711841] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 6680.640496] Key type lgssc registered [ 6683.096404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6725.499674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6725.543446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6727.000809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6727.069325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6727.153282] Lustre: lustre-MDT0000: new disk, initializing [ 6727.225438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6727.244215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6731.998842] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6744.294264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6744.395901] Lustre: 144852:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6744.444201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6744.446929] Lustre: Skipped 1 previous similar message [ 6744.501219] Lustre: lustre-MDT0001: new disk, initializing [ 6744.611991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6744.655250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6744.662124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6749.875577] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6759.350841] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6764.801112] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6773.023287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6773.849527] Lustre: lustre-OST0000: new disk, initializing [ 6773.869287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6774.064697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6780.613473] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6782.548548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6782.555031] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6782.667559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6788.497347] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6794.778984] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_module obdecho/obdecho [ 6796.843858] LustreError: 147469:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6796.853641] LustreError: 147469:0:(obd_config.c:843:class_setup()) setup ec failed (-22) [ 6799.138651] LustreError: 147565:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 6811.111689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6811.131495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6811.161604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6813.668579] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6813.674256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6813.687772] Lustre: Skipped 1 previous similar message [ 6813.697828] Lustre: Skipped 2 previous similar messages [ 6816.973692] Lustre: server umount lustre-MDT0000 complete [ 6820.912354] LustreError: 144845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604818 with bad export cookie 16313866525257272195 [ 6820.921689] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6820.928775] LustreError: 144845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6821.452131] Lustre: server umount lustre-MDT0001 complete [ 6834.889117] Lustre: server umount lustre-OST0000 complete [ 6856.579313] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 6869.771793] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 6913.611914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6913.972969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6914.016367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6914.139491] Lustre: lustre-MDT0000: new disk, initializing [ 6914.210724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6914.229212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6920.278372] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6931.365207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6931.468796] Lustre: 151266:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6931.511145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6931.514764] Lustre: Skipped 1 previous similar message [ 6931.572186] Lustre: lustre-MDT0001: new disk, initializing [ 6931.679900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6931.723451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6931.750950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6935.587764] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6943.236376] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6947.763184] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6955.333418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6955.837592] Lustre: lustre-OST0000: new disk, initializing [ 6955.840914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6955.960589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6963.141505] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6965.325480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6965.344368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6965.472312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6971.931910] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6977.353251] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6977.605411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6982.660472] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6983.005110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6986.208836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6986.214041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6986.221691] LustreError: Skipped 1 previous similar message [ 6986.225704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6986.258240] Lustre: Skipped 1 previous similar message [ 6991.338503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6991.355549] Lustre: Skipped 2 previous similar messages [ 6991.603649] Lustre: server umount lustre-OST0000 complete [ 6996.456603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6996.460941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6996.470216] Lustre: Skipped 1 previous similar message [ 6996.484360] Lustre: Skipped 1 previous similar message [ 7002.607626] Lustre: server umount lustre-MDT0000 complete [ 7006.700031] LustreError: 151277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7006.730141] LustreError: 151277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7007.029384] LustreError: 151258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605004 with bad export cookie 16313866525257273154 [ 7007.041980] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7007.046016] LustreError: 151258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7007.390691] Lustre: server umount lustre-MDT0001 complete [ 7011.981986] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:43:27 (1788605007) [ 7013.874600] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7016.124715] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:43:32 (1788605012) [ 7024.185758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.762447] LustreError: 155272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7024.837580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7029.241459] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7030.241993] LustreError: 155273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7035.362178] LustreError: 155272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7039.285654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.016323] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7050.266708] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7054.253750] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7062.645545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7063.057506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7063.063467] Lustre: Skipped 1 previous similar message [ 7070.030373] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7076.852587] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7082.958503] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.34@tcp (stopping) [ 7082.964411] Lustre: Skipped 2 previous similar messages [ 7084.621960] Lustre: server umount lustre-MDT0000 complete [ 7085.542509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7085.549720] LustreError: 157428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7085.557724] Lustre: Skipped 2 previous similar messages [ 7085.594447] LustreError: 157428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7088.879538] LustreError: 155252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605086 with bad export cookie 16313866525257274036 [ 7088.901809] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7093.664241] LustreError: 155267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7093.687840] LustreError: 155267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7095.365078] Lustre: server umount lustre-MDT0001 complete [ 7101.819583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7116.983949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7117.358576] LustreError: 158780:0:(llog.c:1646:llog_backup()) MGC192.168.201.134@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7117.364230] Lustre: 158780:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.201.134@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7119.329156] LustreError: 158785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.144130] LustreError: 158786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.169665] LustreError: 158786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 7178.208539] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7178.214656] Lustre: 158780:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.201.134@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7219.188607] LustreError: 158787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7219.202404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7219.225432] LustreError: 158787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 42 previous similar messages [ 7224.293158] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7224.775708] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7236.190690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7249.909270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7264.751843] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7264.853831] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7271.604942] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7273.355502] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7277.719393] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7279.646244] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7300.580154] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7300.584732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7300.601042] Lustre: Skipped 2 previous similar messages [ 7300.623070] Lustre: Skipped 5 previous similar messages [ 7302.609516] Lustre: server umount lustre-MDT0000 complete [ 7306.506325] LustreError: 155254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605303 with bad export cookie 16313866525257280511 [ 7306.524291] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7306.531362] LustreError: 155254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7307.122453] Lustre: server umount lustre-MDT0001 complete [ 7321.784757] Lustre: server umount lustre-OST0000 complete [ 7344.580803] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 7356.653601] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 7394.714879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7395.029564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7395.084619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7395.168781] Lustre: lustre-MDT0000: new disk, initializing [ 7395.293409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7395.301952] Lustre: Skipped 1 previous similar message [ 7395.329596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7399.947892] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7410.584807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7410.723215] Lustre: 163636:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7411.080478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7415.791742] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7423.520791] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7427.306853] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7433.799681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7434.017234] Lustre: lustre-OST0000: new disk, initializing [ 7434.022840] Lustre: Skipped 1 previous similar message [ 7434.030469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7434.037939] Lustre: Skipped 2 previous similar messages [ 7440.461590] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7442.471987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7442.478832] Lustre: Skipped 1 previous similar message [ 7442.488963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7442.615865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7448.028441] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7452.991239] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7453.235238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7458.410962] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7458.611838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7463.397278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7463.412916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.435639] Lustre: Skipped 1 previous similar message [ 7463.457390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7466.742089] Lustre: server umount lustre-OST0000 complete [ 7472.608728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7472.619286] LustreError: Skipped 1 previous similar message [ 7476.609741] Lustre: server umount lustre-MDT0000 complete [ 7478.754397] LustreError: 163642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7478.782165] LustreError: 163642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 37 previous similar messages [ 7481.169792] LustreError: 165380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605478 with bad export cookie 16313866525257297325 [ 7481.173524] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7481.180481] LustreError: 165380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7481.578743] Lustre: server umount lustre-MDT0001 complete [ 7492.253232] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:51:28 (1788605488) [ 7498.288710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7502.996725] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7512.662357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7518.356803] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7524.999729] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7528.892434] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7537.014352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7544.046462] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7550.022436] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7560.300142] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:52:36 (1788605556) [ 7564.256403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7564.263578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7564.275763] Lustre: Skipped 3 previous similar messages [ 7564.287949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7564.290675] Lustre: Skipped 4 previous similar messages [ 7570.051869] Lustre: server umount lustre-OST0000 complete [ 7580.878989] Lustre: server umount lustre-MDT0000 complete [ 7584.512932] LustreError: 167625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605581 with bad export cookie 16313866525257298207 [ 7584.523408] LustreError: 167625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7584.534132] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7585.005832] Lustre: server umount lustre-MDT0001 complete [ 7597.025914] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 7600.326850] Key type lgssc unregistered [ 7600.787022] LNet: 171017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7600.800608] LNetError: 171017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7600.852155] LNet: Removed LNI 192.168.201.134@tcp [ 7602.069868] Key type .llcrypt unregistered [ 7602.076324] Key type ._llcrypt unregistered [ 7626.394441] Key type ._llcrypt registered [ 7626.399310] Key type .llcrypt registered [ 7626.539257] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 7628.158627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7628.327373] alg: No test for adler32 (adler32-zlib) [ 7629.409499] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 7629.660231] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 7631.375229] Key type lgssc registered [ 7632.548180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7641.036536] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7641.068735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7656.735954] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7661.855253] LustreError: 172249:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.134@tcp: failed processing log, type 4: rc = -110 [ 7666.975899] LustreError: 172249:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7667.007645] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7667.016225] LustreError: MGC192.168.201.134@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7667.038820] LustreError: 172249:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7667.056362] LustreError: 172249:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -22 [ 7667.066589] LustreError: 172249:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 7667.124564] Lustre: server umount lustre-OST0000 complete [ 7667.134139] LustreError: 172249:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7676.369435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7677.957224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7682.405529] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7688.178983] LustreError: 172748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7688.207122] LustreError: 172748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7692.226530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7692.556088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7697.356909] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7703.693093] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7707.884520] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7716.090236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7716.312603] LustreError: 174337:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7716.326891] Lustre: 174337:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.201.134@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7716.479690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7723.361429] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7731.725761] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7738.850069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7738.856360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7738.875873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7739.372750] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7739.376508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7739.386679] Lustre: Skipped 1 previous similar message [ 7739.409861] Lustre: Skipped 1 previous similar message [ 7741.582770] Lustre: server umount lustre-MDT0000 complete [ 7744.483564] LustreError: 173862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7744.507952] LustreError: 173862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7746.442918] LustreError: 172730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605743 with bad export cookie 11030248041370890182 [ 7746.455876] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7746.696815] Lustre: server umount lustre-MDT0001 complete [ 7761.217618] Lustre: server umount lustre-OST0000 complete [ 7771.232850] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 7780.823580] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 7817.961963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7818.199279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7818.247174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7818.331907] Lustre: lustre-MDT0000: new disk, initializing [ 7818.501581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7818.532301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7824.725915] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7837.611029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7837.721656] Lustre: 177721:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7837.761853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7837.765345] Lustre: Skipped 1 previous similar message [ 7837.824810] Lustre: lustre-MDT0001: new disk, initializing [ 7837.911115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7837.946851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7837.955836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7843.616284] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7852.063634] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7856.578943] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7864.027228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7864.223620] Lustre: lustre-OST0000: new disk, initializing [ 7864.226322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7864.320044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7870.852995] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7871.059804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7871.068045] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7871.181145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7878.096590] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7882.893340] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7883.132693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7888.377167] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7888.622533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7891.937365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7891.943802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7891.948057] LustreError: Skipped 1 previous similar message [ 7891.951319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7891.959964] Lustre: Skipped 1 previous similar message [ 7891.984206] Lustre: Skipped 2 previous similar messages [ 7897.154490] Lustre: server umount lustre-OST0000 complete [ 7902.179090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7902.199843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7902.205896] Lustre: Skipped 1 previous similar message [ 7902.224162] Lustre: Skipped 1 previous similar message [ 7907.297118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7907.306709] Lustre: Skipped 1 previous similar message [ 7908.267303] Lustre: server umount lustre-MDT0000 complete [ 7912.270784] LustreError: 177711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605909 with bad export cookie 11030248041370890896 [ 7912.271771] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7912.288792] LustreError: 177711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7912.601666] Lustre: server umount lustre-MDT0001 complete [ 7921.362892] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:58:37 (1788605917) [ 7937.877212] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 7940.381307] Key type lgssc unregistered [ 7940.676539] LNet: 182245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7940.689437] LNetError: 182245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7940.702818] LNet: Removed LNI 192.168.201.134@tcp [ 7941.437685] Key type .llcrypt unregistered [ 7941.439365] Key type ._llcrypt unregistered [ 7959.012320] Key type ._llcrypt registered [ 7959.014854] Key type .llcrypt registered [ 7959.110869] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 7959.981178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7960.074319] alg: No test for adler32 (adler32-zlib) [ 7961.133405] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 7961.347526] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 7963.023232] Key type lgssc registered [ 7964.503503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7972.565488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7972.579979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7974.199851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7978.054201] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7984.619988] LustreError: 183449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7984.630437] LustreError: 183449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7986.792940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7987.120505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7990.904248] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 7997.116402] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8001.362730] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8010.689367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8010.937398] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8010.942308] LustreError: 185085:0:(llog.c:1687:llog_backup()) MGC192.168.201.134@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8010.951117] Lustre: 185085:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.201.134@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8011.054619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8018.140299] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8027.251949] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8030.693626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8030.709196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8030.716747] Lustre: Skipped 2 previous similar messages [ 8030.750932] Lustre: Skipped 2 previous similar messages [ 8035.812617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8035.836777] Lustre: Skipped 2 previous similar messages [ 8037.017353] Lustre: server umount lustre-MDT0000 complete [ 8040.935185] LustreError: 183448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8040.956533] LustreError: 183448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8041.042252] LustreError: 183428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606038 with bad export cookie 2937652316455696569 [ 8041.063325] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8041.538786] Lustre: server umount lustre-MDT0001 complete [ 8054.970471] Lustre: server umount lustre-OST0000 complete [ 8065.213696] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 8074.755979] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 8117.750901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8118.038561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8118.084403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8118.192637] Lustre: lustre-MDT0000: new disk, initializing [ 8118.264225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8118.288722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8124.095630] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8136.326298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8136.445650] Lustre: 188481:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8136.488455] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8136.490268] Lustre: Skipped 1 previous similar message [ 8136.492846] LustreError: 189246:0:(llog.c:1687:llog_backup()) MGC192.168.201.134@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8136.498389] LustreError: 189246:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8136.502215] Lustre: 189246:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.201.134@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8136.517661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8136.520944] Lustre: Skipped 1 previous similar message [ 8136.601705] Lustre: lustre-MDT0001: new disk, initializing [ 8136.727244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8136.761702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8136.773337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8141.441360] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8149.921210] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8155.072481] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8162.585870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8162.942702] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8162.944383] Lustre: Skipped 1 previous similar message [ 8162.947025] LustreError: 190197:0:(llog.c:1687:llog_backup()) MGC192.168.201.134@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8162.952384] LustreError: 190197:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8162.956264] Lustre: 190197:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.201.134@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8162.962016] Lustre: 190197:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8162.984789] Lustre: lustre-OST0000: new disk, initializing [ 8162.988993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8163.091833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8169.611248] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8170.059096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8170.067888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8170.192813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8177.338462] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8182.113445] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8182.439894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8187.460619] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8187.711600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8190.954522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8190.970329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8190.982164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8196.078035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8196.092472] Lustre: Skipped 2 previous similar messages [ 8201.186914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8201.193973] Lustre: Skipped 1 previous similar message [ 8204.255309] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8204.454275] Lustre: server umount lustre-OST0000 complete [ 8208.352286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8208.364477] LustreError: Skipped 1 previous similar message [ 8208.374931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.396607] Lustre: Skipped 1 previous similar message [ 8211.426381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8211.434994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8211.442438] Lustre: Skipped 1 previous similar message [ 8214.480671] Lustre: server umount lustre-MDT0000 complete [ 8216.544535] LustreError: 188486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8216.572209] LustreError: 188486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8218.254691] LustreError: 188473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606215 with bad export cookie 2937652316455697290 [ 8218.262880] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8218.286128] LustreError: 188473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8218.747635] Lustre: server umount lustre-MDT0001 complete [ 8227.406704] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 07:03:43 (1788606223) [ 8243.776935] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 8253.033369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8253.312604] LustreError: 193188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8253.385591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8257.671591] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8258.539891] LustreError: 193189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8266.515295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8271.467888] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8274.453767] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8280.710686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8288.303681] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8297.451480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8297.612828] Lustre: lustre-OST0001: new disk, initializing [ 8297.616662] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8297.754925] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8297.763680] Lustre: Skipped 2 previous similar messages [ 8303.857575] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8307.837733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8307.852669] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8308.024272] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8315.459707] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8322.282629] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8329.050917] Lustre: DEBUG MARKER: ADD OST3 [ 8333.967318] loop: module loaded [ 8334.049870] loop0: detected capacity change from 0 to 204800000 [ 8334.302208] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8334.323051] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8334.353355] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8340.617983] loop0: detected capacity change from 0 to 204800000 [ 8341.334892] Lustre: DEBUG MARKER: STOP OST3 [ 8347.004763] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8347.246567] Lustre: lustre-OST0003: new disk, initializing [ 8347.252039] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8348.718588] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8348.888536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8348.898289] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8348.938695] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8350.757644] LustreError: 197590:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8351.250947] Lustre: Failing over lustre-OST0003 [ 8351.373920] Lustre: server umount lustre-OST0003 complete [ 8353.790672] LustreError: 194643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8353.805865] LustreError: 194643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8354.291137] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8354.981430] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8363.650745] Lustre: DEBUG MARKER: START OST3 again [ 8371.102711] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8371.473665] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8371.482948] Lustre: Skipped 1 previous similar message [ 8371.496953] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8372.737852] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8373.358388] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8373.361720] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8377.933475] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8399.826906] Lustre: Failing over lustre-OST0003 [ 8400.136749] Lustre: server umount lustre-OST0003 complete [ 8402.419739] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8402.436632] LustreError: 195926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8402.443321] Lustre: Skipped 2 previous similar messages [ 8402.485409] LustreError: 195926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 8420.834265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8420.845995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8420.857047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8420.865027] Lustre: Skipped 1 previous similar message [ 8424.903777] Lustre: server umount lustre-MDT0000 complete [ 8429.500674] LustreError: 196199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606426 with bad export cookie 2937652316455698172 [ 8429.520108] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8429.522651] LustreError: 196199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8436.113656] Lustre: server umount lustre-MDT0001 complete [ 8440.426444] Lustre: server umount lustre-OST0000 complete [ 8445.357709] Lustre: server umount lustre-OST0001 complete [ 8452.994331] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 8462.830952] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 8503.422308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8503.830223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8503.913717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8504.087329] Lustre: lustre-MDT0000: new disk, initializing [ 8504.245241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8504.293639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8510.064242] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8523.660435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8523.815243] Lustre: 202825:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8523.881213] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8523.888424] Lustre: Skipped 1 previous similar message [ 8523.989640] Lustre: lustre-MDT0001: new disk, initializing [ 8524.218108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8524.249246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8529.427315] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8536.772897] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8540.479863] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8547.388030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8549.582971] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8549.694762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8554.769906] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8563.139836] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8568.245846] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8568.514213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8573.171276] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8573.422063] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8575.458620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8575.465525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8575.489927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8575.515438] Lustre: Skipped 6 previous similar messages [ 8575.534504] Lustre: Skipped 8 previous similar messages [ 8581.633870] Lustre: server umount lustre-OST0000 complete [ 8590.816657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8592.400247] Lustre: server umount lustre-MDT0000 complete [ 8595.937718] LustreError: 204484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8595.954971] LustreError: 204484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 8596.832908] LustreError: 202818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606594 with bad export cookie 2937652316455700468 [ 8596.842101] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8596.845094] LustreError: 202818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8597.268149] Lustre: server umount lustre-MDT0001 complete [ 8601.109534] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 07:09:57 (1788606597) [ 8617.327904] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 8627.727953] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 8667.197627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8667.489528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8667.517664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8667.520979] Lustre: Skipped 1 previous similar message [ 8667.686415] Lustre: lustre-MDT0000: new disk, initializing [ 8667.689983] Lustre: Skipped 1 previous similar message [ 8667.848604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8667.855734] Lustre: Skipped 1 previous similar message [ 8673.495842] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8686.715463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8686.805764] Lustre: 208997:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8687.142409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8691.849375] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8702.784472] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8707.770091] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8717.101377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8718.822126] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8724.223235] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8733.139519] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8737.794300] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8738.051351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8742.380087] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8742.587393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8749.541985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8749.551050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8749.567299] LustreError: Skipped 1 previous similar message [ 8749.573582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8749.586468] Lustre: Skipped 3 previous similar messages [ 8749.606265] Lustre: Skipped 5 previous similar messages [ 8752.176542] Lustre: server umount lustre-OST0000 complete [ 8763.319320] Lustre: server umount lustre-MDT0000 complete [ 8764.899735] LustreError: 209010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8764.932674] LustreError: 209010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8768.295262] LustreError: 208991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606765 with bad export cookie 2937652316455701343 [ 8768.307892] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8768.313815] LustreError: 208991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8768.749899] Lustre: server umount lustre-MDT0001 complete [ 8777.116691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8777.427831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8777.432551] Lustre: Skipped 5 previous similar messages [ 8782.089589] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8790.753036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8795.577940] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8800.830307] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8804.286550] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8811.422306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8818.124825] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 8823.762804] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8836.581098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8836.587037] LustreError: Skipped 1 previous similar message [ 8839.943680] Lustre: server umount lustre-OST0000 complete [ 8851.300263] Lustre: server umount lustre-MDT0000 complete [ 8855.759951] LustreError: 212797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606853 with bad export cookie 2937652316455702232 [ 8855.765568] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8855.772370] LustreError: 212797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8856.234795] Lustre: server umount lustre-MDT0001 complete [ 8962.342507] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 8965.129624] Key type lgssc unregistered [ 8965.468181] LNet: 216158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8965.476412] LNetError: 216158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8965.502417] LNet: Removed LNI 192.168.201.134@tcp [ 8966.453160] Key type .llcrypt unregistered [ 8966.458976] Key type ._llcrypt unregistered [ 8992.703242] Key type ._llcrypt registered [ 8992.704619] Key type .llcrypt registered [ 8992.835827] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 9009.185942] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 9010.465461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9010.477382] alg: No test for adler32 (adler32-zlib) [ 9011.705411] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 9011.958045] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 9013.727497] Key type lgssc registered [ 9014.731870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9049.061310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9049.083629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9050.368704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9050.426304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9050.539341] Lustre: lustre-MDT0000: new disk, initializing [ 9050.732597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9050.757956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9055.091446] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9066.403146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9066.544732] Lustre: 219476:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9066.627436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9066.631414] Lustre: Skipped 1 previous similar message [ 9066.721149] Lustre: lustre-MDT0001: new disk, initializing [ 9066.806367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9066.865101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9066.876895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9072.021553] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9079.970563] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9083.882811] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9090.237173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9090.588037] Lustre: lustre-OST0000: new disk, initializing [ 9090.591529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9090.668883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9096.340171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9096.356645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9096.491496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9097.419110] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9105.832499] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9110.953564] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9111.404538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9116.216521] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9116.633186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9122.276950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9122.280144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9122.293417] LustreError: Skipped 1 previous similar message [ 9122.295515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9122.305308] Lustre: Skipped 1 previous similar message [ 9125.474563] Lustre: server umount lustre-OST0000 complete [ 9132.517103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9132.537365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9132.551854] Lustre: Skipped 2 previous similar messages [ 9136.005167] Lustre: server umount lustre-MDT0000 complete [ 9139.903096] LustreError: 219469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788607137 with bad export cookie 2842752750720336940 [ 9139.918499] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9139.922092] LustreError: 219469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9140.376650] Lustre: server umount lustre-MDT0001 complete [ 9159.112670] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 9170.007959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9170.418315] LustreError: 223987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9170.435372] LustreError: 223987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9170.467234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9175.134032] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9175.523530] LustreError: 223988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9179.617095] LustreError: 223987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9184.478189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9184.856444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9189.528268] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9192.557058] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9198.589245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9198.832547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9205.594555] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9215.672729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9215.933746] Lustre: lustre-OST0001: new disk, initializing [ 9215.944136] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9216.084532] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9222.694453] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9224.413340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9224.428588] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9224.574777] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9237.375469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9244.203128] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9262.777594] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 07:20:59 (1788607259) [ 9265.634124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9265.649744] Lustre: Skipped 1 previous similar message [ 9265.651322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9265.683963] Lustre: Skipped 3 previous similar messages [ 9270.755776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9270.776635] Lustre: Skipped 3 previous similar messages [ 9271.926982] Lustre: server umount lustre-MDT0000 complete [ 9275.252626] LustreError: 223967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788607272 with bad export cookie 2842752750720337822 [ 9275.266478] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9275.269353] LustreError: 223967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9275.841827] Lustre: server umount lustre-MDT0001 complete [ 9290.611938] Lustre: server umount lustre-OST0000 complete [ 9291.746671] Lustre: 217701:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788607273/real 1788607273] req@ffff9f45487e5180 x1875490383182976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788607289 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9291.818487] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9291.837520] Lustre: Skipped 2 previous similar messages [ 9296.073931] Lustre: server umount lustre-OST0001 complete [ 9303.506574] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 9315.066359] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 9351.513751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9351.868221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9351.903112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9352.033974] Lustre: lustre-MDT0000: new disk, initializing [ 9352.137201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9352.160294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9356.515519] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9366.881592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9366.964440] Lustre: 230167:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9367.015259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9367.020241] Lustre: Skipped 1 previous similar message [ 9367.120782] Lustre: lustre-MDT0001: new disk, initializing [ 9367.301155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9367.318683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9372.158467] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9380.312848] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9384.321327] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9391.493194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9391.906781] Lustre: lustre-OST0000: new disk, initializing [ 9391.911232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9393.139902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9393.162940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9393.380268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9398.881238] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9407.103821] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9412.160313] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9412.477518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9417.280486] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9417.518446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9423.849878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9423.857431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9423.866804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9426.332864] Lustre: server umount lustre-OST0000 complete [ 9434.081201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9434.085916] LustreError: Skipped 1 previous similar message [ 9434.089088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9434.096453] Lustre: Skipped 1 previous similar message [ 9434.100049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9434.104439] Lustre: Skipped 2 previous similar messages [ 9445.345557] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9445.691430] Lustre: server umount lustre-MDT0000 complete [ 9448.932751] LustreError: 230179:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9448.956726] LustreError: 230179:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9450.458593] LustreError: 230945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788607447 with bad export cookie 2842752750720339964 [ 9450.471806] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9450.478448] LustreError: 230945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9450.887805] Lustre: server umount lustre-MDT0001 complete [ 9460.720276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9461.372747] LustreError: 233980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9461.471673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9461.479719] Lustre: Skipped 2 previous similar messages [ 9466.787644] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9470.968247] LustreError: 233980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9470.986507] LustreError: 233980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9476.488676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9481.761151] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9487.198635] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9491.415976] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9497.908764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9504.470821] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9510.730621] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9659.875304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9659.877956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9659.895837] Lustre: Skipped 2 previous similar messages [ 9659.918062] Lustre: Skipped 6 previous similar messages [ 9665.273898] Lustre: server umount lustre-OST0000 complete [ 9677.838882] Lustre: server umount lustre-MDT0000 complete [ 9680.355174] LustreError: 233980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9680.382415] LustreError: 233980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9681.744602] LustreError: 233962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788607679 with bad export cookie 2842752750720340846 [ 9681.752941] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9682.202462] Lustre: server umount lustre-MDT0001 complete [ 9779.421682] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 9782.611174] Key type lgssc unregistered [ 9782.922835] LNet: 237233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9782.933885] LNetError: 237233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9782.951800] LNet: Removed LNI 192.168.201.134@tcp [ 9784.212182] Key type .llcrypt unregistered [ 9784.214091] Key type ._llcrypt unregistered [ 9815.177145] Key type ._llcrypt registered [ 9815.178905] Key type .llcrypt registered [ 9815.340994] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 9830.740453] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 9832.156292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9832.180161] alg: No test for adler32 (adler32-zlib) [ 9833.255664] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 9833.450460] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 9835.167403] Key type lgssc registered [ 9836.452188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9883.806297] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 9895.012843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9895.047899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9896.433673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9896.467427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9896.599607] Lustre: lustre-MDT0000: new disk, initializing [ 9896.673339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9896.689715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9901.794192] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9913.397947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9913.570283] Lustre: 241253:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9913.620748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9913.623961] Lustre: Skipped 1 previous similar message [ 9913.676905] Lustre: lustre-MDT0001: new disk, initializing [ 9913.756665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9913.782197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9913.798398] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9918.700540] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9923.837290] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9931.147714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9931.384557] Lustre: lustre-OST0000: new disk, initializing [ 9931.388291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9931.478325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9933.407640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9933.423400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9933.610891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9937.666876] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9948.118198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9948.311557] Lustre: lustre-OST0001: new disk, initializing [ 9948.320338] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9948.449106] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9955.007607] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 9956.995173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9957.012879] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9957.204766] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9968.121434] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9975.418949] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9993.188541] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9993.214513] Lustre: Skipped 3 previous similar messages [ 9994.943269] Lustre: server umount lustre-MDT0000 complete [ 9998.314819] LustreError: 241264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9998.337043] LustreError: 241264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [10000.005129] LustreError: 241243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788607997 with bad export cookie 7946507203338640991 [10000.022156] LustreError: 241243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10000.026764] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10000.643923] Lustre: server umount lustre-MDT0001 complete [10013.988917] Lustre: server umount lustre-OST0000 complete [10029.446662] Lustre: server umount lustre-OST0001 complete [10046.097713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10056.175430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10065.836754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10076.787377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10097.814787] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [10107.561521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10107.576103] Lustre: lustre-MDT0000: reset Object Index mappings [10108.023809] LustreError: 248685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10108.114751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10112.971692] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10113.512463] LustreError: 248686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10118.633249] LustreError: 248685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10122.381202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10122.415243] Lustre: lustre-MDT0001: reset Object Index mappings [10123.062060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10127.602754] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10131.030353] Lustre: 249793:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10139.143308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10139.557702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10146.501787] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10147.763391] LustreError: 250145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10153.982188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10155.684717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10157.637512] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10163.967382] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10173.491535] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10177.567893] Lustre: 251639:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10189.794540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10189.802597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10189.825586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10190.071337] Lustre: server umount lustre-MDT0000 complete [10191.844450] LustreError: 248686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10191.848755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10191.857682] LustreError: 248686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10191.886868] Lustre: Skipped 1 previous similar message [10194.100556] LustreError: 248666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788608191 with bad export cookie 7946507203338647781 [10194.108232] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10194.109200] LustreError: 248666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10194.427843] Lustre: server umount lustre-MDT0001 complete [10208.904142] Lustre: server umount lustre-OST0000 complete [10212.831802] Lustre: 238779:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788608194/real 1788608194] req@ffff9f467d18bb80 x1875491244087296/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788608210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10212.864845] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10212.876439] Lustre: Skipped 1 previous similar message [10213.165365] Lustre: server umount lustre-OST0001 complete [10233.113965] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:37:09 (1788608229) [10250.879828] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [10261.995833] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [10301.919637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10302.278579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10302.326205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10302.470182] Lustre: lustre-MDT0000: new disk, initializing [10302.600085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10302.609500] Lustre: Skipped 1 previous similar message [10302.623725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10307.410880] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10319.063408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10319.322494] Lustre: 255864:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10319.458967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10319.470258] Lustre: Skipped 1 previous similar message [10319.611505] Lustre: lustre-MDT0001: new disk, initializing [10319.878533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10319.893812] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10324.834280] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10332.604632] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10337.501732] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10345.271902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10345.573903] Lustre: lustre-OST0000: new disk, initializing [10345.577475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10347.376499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10347.395651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10347.561462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10352.920661] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10361.799639] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10367.075174] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10367.332457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10372.727158] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10372.928286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10378.226530] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10378.232586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10378.259092] LustreError: Skipped 1 previous similar message [10378.281386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10378.294659] Lustre: Skipped 1 previous similar message [10381.466862] Lustre: server umount lustre-OST0000 complete [10386.401713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10386.411120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10386.432225] Lustre: Skipped 1 previous similar message [10386.449673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10388.451496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10391.997836] Lustre: server umount lustre-MDT0000 complete [10393.569087] LustreError: 255869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10393.587632] LustreError: 255869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10396.440792] LustreError: 256643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788608393 with bad export cookie 7946507203338649874 [10396.443666] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10396.452635] LustreError: 256643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10396.871332] Lustre: server umount lustre-MDT0001 complete [10418.738941] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [10432.101307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10432.693258] LustreError: 260378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10432.773286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10432.780968] Lustre: Skipped 2 previous similar messages [10438.429737] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10448.895858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10454.004286] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10457.492881] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10464.797308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10471.015319] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10479.669611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10479.840224] Lustre: lustre-OST0001: new disk, initializing [10479.844540] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10481.464486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10481.473789] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10481.601826] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10486.344345] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10496.480446] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10500.223778] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10511.239577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10511.287674] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10517.479760] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10517.487739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10517.498700] Lustre: Skipped 3 previous similar messages [10517.517410] Lustre: Skipped 4 previous similar messages [10527.723861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10527.731932] Lustre: Skipped 5 previous similar messages [10531.296073] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10531.565441] Lustre: server umount lustre-MDT0000 complete [10532.836935] LustreError: 263504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10532.867280] LustreError: 263504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10535.494176] LustreError: 262636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788608532 with bad export cookie 7946507203338650756 [10535.499211] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10535.515251] LustreError: 262636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10536.079381] Lustre: server umount lustre-MDT0001 complete [10550.607309] Lustre: server umount lustre-OST0000 complete [10553.825123] Lustre: 238779:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788608535/real 1788608535] req@ffff9f467e99ed80 x1875491244194176/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788608551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10553.888865] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.916966] Lustre: Skipped 1 previous similar message [10554.939103] Lustre: server umount lustre-OST0001 complete [10573.666565] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [10584.729781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10585.161576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10585.168074] Lustre: Skipped 3 previous similar messages [10589.766756] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10599.168989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10604.250640] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10607.265406] Lustre: 266766:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10614.331815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10621.468466] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10631.287142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10631.980997] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10633.944829] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10639.842450] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10648.669966] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10652.720906] Lustre: 268622:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10664.040664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10664.248613] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10674.854421] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:44:30 (1788608670) [10691.552673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10691.557839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10691.576612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10698.212928] LustreError: 265659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10698.237466] LustreError: 265659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [10698.279754] Lustre: server umount lustre-MDT0000 complete [10703.433027] LustreError: 265640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788608700 with bad export cookie 7946507203338657049 [10703.439521] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10703.444882] LustreError: 265640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10710.029691] Lustre: server umount lustre-MDT0001 complete [10720.263443] Lustre: server umount lustre-OST0000 complete [10726.275183] Lustre: server umount lustre-OST0001 complete [10735.079839] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [10746.956487] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [10795.713967] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [10808.961710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10809.395983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10809.463781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10809.582038] Lustre: lustre-MDT0000: new disk, initializing [10809.763550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10814.983331] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10827.461934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10827.650682] Lustre: 272812:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10828.038295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10833.294481] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10845.168344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10845.819996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10845.836218] Lustre: Skipped 5 previous similar messages [10847.279921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10853.640996] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10865.400867] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10867.045270] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10867.060607] Lustre: Skipped 1 previous similar message [10867.298369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10875.364332] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [10889.565796] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10893.351399] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10893.361366] Lustre: Skipped 1 previous similar message [10916.478921] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 07:48:32 (1788608912) [10920.417621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10920.427232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10920.439726] Lustre: Skipped 5 previous similar messages [10920.442646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10920.444815] Lustre: Skipped 6 previous similar messages [10921.313530] Lustre: server umount lustre-MDT0000 complete [10925.729676] LustreError: 272804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788608923 with bad export cookie 7946507203338664924 [10925.738504] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10925.744180] LustreError: 272804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10926.151338] Lustre: server umount lustre-MDT0001 complete [10940.607633] Lustre: server umount lustre-OST0000 complete [10954.574585] Lustre: server umount lustre-OST0001 complete [10971.421303] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [10980.966377] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [11018.966767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11019.309765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11019.365885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11019.373967] Lustre: Skipped 4 previous similar messages [11019.471680] Lustre: lustre-MDT0000: new disk, initializing [11019.476970] Lustre: Skipped 3 previous similar messages [11019.629954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11019.647477] Lustre: Skipped 3 previous similar messages [11024.056103] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11034.593109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11034.723645] Lustre: 279860:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11034.974910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11039.625300] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11047.980720] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11052.662929] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11059.572156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11061.663919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11066.607897] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11073.560072] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11075.539256] Lustre: Mounted lustre-client [11076.418292] LustreError: 282358:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff9f4644ef7800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [11076.527791] Lustre: Unmounted lustre-client [11087.330351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11087.334582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11087.344665] LustreError: Skipped 1 previous similar message [11093.230166] Lustre: server umount lustre-OST0000 complete [11111.391928] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11111.854126] Lustre: server umount lustre-MDT0000 complete [11115.287240] LustreError: 279851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788609112 with bad export cookie 7946507203338666450 [11115.288423] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11115.297347] LustreError: 279851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11115.764382] Lustre: server umount lustre-MDT0001 complete [11127.633777] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [11128.927162] LNet: 1 local NIs in recovery (showing 1): 192.168.201.134@tcp [11131.003301] Key type lgssc unregistered [11131.442728] LNet: 283641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11131.460117] LNetError: 283641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11208.294610] LNet: Removed LNI 192.168.201.134@tcp [11209.060329] Key type .llcrypt unregistered [11209.063242] Key type ._llcrypt unregistered [11229.128800] Key type ._llcrypt registered [11229.138256] Key type .llcrypt registered [11229.313312] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [11251.112697] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [11252.697613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11252.796136] alg: No test for adler32 (adler32-zlib) [11254.062460] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [11254.403755] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [11256.087782] Key type lgssc registered [11257.877945] Lustre: Echo OBD driver; http://www.lustre.org/ [11299.136602] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:54:55 (1788609295) [11307.750987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11307.809617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11309.456378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11309.548888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11309.725231] Lustre: lustre-MDT0000: new disk, initializing [11309.875095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11309.892189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11314.546716] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11324.696856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11324.864588] Lustre: 286826:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11324.923159] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11324.926564] Lustre: Skipped 1 previous similar message [11324.993506] Lustre: lustre-MDT0001: new disk, initializing [11325.084576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11325.116774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11325.127492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11329.548268] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11337.578023] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11341.410398] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11348.177265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11348.421966] Lustre: lustre-OST0000: new disk, initializing [11348.425777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11348.528520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11354.509109] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11355.710020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11355.727694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11355.848285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11362.833722] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11376.608538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11376.615241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11376.623974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11376.626863] Lustre: Skipped 1 previous similar message [11379.562595] Lustre: server umount lustre-OST0000 complete [11386.850752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11386.857034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11386.862154] LustreError: Skipped 1 previous similar message [11386.896744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11386.911804] Lustre: Skipped 2 previous similar messages [11389.563707] Lustre: server umount lustre-MDT0000 complete [11393.702189] LustreError: 286820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788609391 with bad export cookie 7282458366510066091 [11393.708462] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11393.717547] LustreError: 286820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11393.994075] Lustre: server umount lustre-MDT0001 complete [11405.947805] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [11409.568153] Key type lgssc unregistered [11409.855709] LNet: 290203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11409.859610] LNetError: 290203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11409.876956] LNet: Removed LNI 192.168.201.134@tcp [11410.787799] Key type .llcrypt unregistered [11410.792574] Key type ._llcrypt unregistered [11440.024461] Key type ._llcrypt registered [11440.027168] Key type .llcrypt registered [11440.224674] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [11441.273509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11441.461026] alg: No test for adler32 (adler32-zlib) [11442.598307] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [11442.822329] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [11444.465058] Key type lgssc registered [11445.895445] Lustre: Echo OBD driver; http://www.lustre.org/ [11454.178980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11454.212839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11455.619615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11460.436862] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11465.188437] LustreError: 291603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11465.214283] LustreError: 291603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11470.304388] LustreError: 291602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11471.417530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11471.994760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11477.878677] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11485.885028] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11489.970022] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11497.364372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11497.882996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11505.177807] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11512.146852] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11523.300766] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11525.308481] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11532.771445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11532.774788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11532.787819] Lustre: Skipped 2 previous similar messages [11532.792611] Lustre: Skipped 1 previous similar message [11534.295435] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.34@tcp (stopping) [11534.302245] Lustre: Skipped 1 previous similar message [11535.819754] Lustre: server umount lustre-MDT0000 complete [11537.904376] LustreError: 291603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11537.928850] LustreError: 291603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11539.939680] LustreError: 291584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788609537 with bad export cookie 10157432718674406543 [11539.946509] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11539.954268] LustreError: 291584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11543.012232] LustreError: 291598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11543.013983] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11543.038527] LustreError: 291598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11543.067368] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11546.573348] Lustre: server umount lustre-MDT0001 complete [11551.619503] Lustre: server umount lustre-OST0000 complete [11562.558051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11569.025394] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11573.675834] Lustre: 295335:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.134@tcp [11576.484040] Lustre: server umount MGS complete [11587.085317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11587.556602] LustreError: 295907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11587.576352] LustreError: 295907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11587.630856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11592.977993] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11596.769744] LustreError: 295908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11596.806696] LustreError: 295908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11603.542885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11603.967646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11609.382476] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11619.670670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11619.914951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11627.642904] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11634.142242] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11636.239471] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11639.928622] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11641.602464] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11647.969725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11647.971614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11647.978381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11647.978390] Lustre: Skipped 2 previous similar messages [11647.979840] LustreError: Skipped 1 previous similar message [11648.033205] Lustre: Skipped 1 previous similar message [11652.135534] Lustre: server umount lustre-OST0000 complete [11658.211968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11658.215896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11658.228910] Lustre: Skipped 1 previous similar message [11658.240107] Lustre: Skipped 2 previous similar messages [11662.761524] Lustre: server umount lustre-MDT0000 complete [11663.333902] LustreError: 295908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11663.363379] LustreError: 295908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11667.043626] LustreError: 295889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788609664 with bad export cookie 10157432718674407383 [11667.053909] LustreError: 295889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11667.065820] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11667.542876] Lustre: server umount lustre-MDT0001 complete [11677.803878] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [11681.391497] Key type lgssc unregistered [11681.818824] LNet: 299098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11681.824650] LNetError: 299098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11681.859088] LNet: Removed LNI 192.168.201.134@tcp [11683.054359] Key type .llcrypt unregistered [11683.056398] Key type ._llcrypt unregistered [11704.805973] Key type ._llcrypt registered [11704.808183] Key type .llcrypt registered [11704.935239] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [11724.035891] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [11725.778441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11725.796642] alg: No test for adler32 (adler32-zlib) [11726.948070] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [11727.260339] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [11729.047211] Key type lgssc registered [11730.509915] Lustre: Echo OBD driver; http://www.lustre.org/ [11771.543490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11771.576871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11772.913605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11772.988557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11773.097597] Lustre: lustre-MDT0000: new disk, initializing [11773.224153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11773.250743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11778.954606] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11789.818372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11789.938897] Lustre: 302221:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11789.987813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11789.991600] Lustre: Skipped 1 previous similar message [11790.061986] Lustre: lustre-MDT0001: new disk, initializing [11790.152618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11790.188384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11790.208717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11794.662392] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11803.021839] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11806.879424] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11813.803876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11814.134415] Lustre: lustre-OST0000: new disk, initializing [11814.140680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11814.263604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11820.574488] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11822.208827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11822.230449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11822.372801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11828.395266] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11833.560275] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11833.756520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11840.401368] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11840.671494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11848.181256] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11848.208217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11848.235723] Lustre: Skipped 1 previous similar message [11848.241044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11848.265371] Lustre: Skipped 1 previous similar message [11853.286166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11853.301153] Lustre: Skipped 1 previous similar message [11857.375916] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11857.611345] Lustre: server umount lustre-OST0000 complete [11863.524188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11863.539995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11863.544906] Lustre: Skipped 1 previous similar message [11863.566448] Lustre: Skipped 1 previous similar message [11868.639820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11869.178475] Lustre: server umount lustre-MDT0000 complete [11873.419540] LustreError: 302216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788609870 with bad export cookie 10004627171605219479 [11873.424602] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11873.429038] LustreError: 302216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11874.467904] Lustre: server umount lustre-MDT0001 complete [11887.807473] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 08:04:42 (1788609882) [11909.761720] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [11922.237714] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [11962.323505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11962.462792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11962.486120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11962.545261] Lustre: lustre-MDT0000: new disk, initializing [11962.612356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11962.632615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11966.869467] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11978.806435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11978.930966] Lustre: 308545:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11978.984693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11978.989174] Lustre: Skipped 1 previous similar message [11979.065324] Lustre: lustre-MDT0001: new disk, initializing [11979.183369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11979.222622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11979.235385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11983.624825] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [11991.583730] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11996.489539] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12005.631917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12006.092386] Lustre: lustre-OST0000: new disk, initializing [12006.099922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12006.323688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12015.036900] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12015.684450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12015.696255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12015.855533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12024.286651] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12067.302259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12067.313131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12067.327977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12067.337218] LustreError: Skipped 1 previous similar message [12067.364689] Lustre: Skipped 2 previous similar messages [12070.162672] Lustre: server umount lustre-OST0000 complete [12076.512601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12076.521156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12076.531418] Lustre: Skipped 1 previous similar message [12076.539731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12081.146469] Lustre: server umount lustre-MDT0000 complete [12084.822159] LustreError: 308535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788610082 with bad export cookie 10004627171605220368 [12084.825196] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12084.841079] LustreError: 308535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12085.218799] Lustre: server umount lustre-MDT0001 complete [12098.633331] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12102.084131] Key type lgssc unregistered [12102.537837] LNet: 312331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12102.556242] LNetError: 312331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12103.595241] LNet: Removed LNI 192.168.201.134@tcp [12104.769647] Key type .llcrypt unregistered [12104.772580] Key type ._llcrypt unregistered [12208.595365] Key type ._llcrypt registered [12208.599254] Key type .llcrypt registered [12208.757047] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12209.272404] Key type .llcrypt unregistered [12209.276674] Key type ._llcrypt unregistered [12299.211825] Key type ._llcrypt registered [12299.215157] Key type .llcrypt registered [12299.309388] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12299.568730] Key type .llcrypt unregistered [12299.571267] Key type ._llcrypt unregistered [12332.418623] Key type ._llcrypt registered [12332.420313] Key type .llcrypt registered [12332.586804] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12332.955402] Key type .llcrypt unregistered [12332.958377] Key type ._llcrypt unregistered [12366.359244] Key type ._llcrypt registered [12366.363291] Key type .llcrypt registered [12366.449304] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12366.711378] Key type .llcrypt unregistered [12366.713238] Key type ._llcrypt unregistered [12454.360162] Key type ._llcrypt registered [12454.365726] Key type .llcrypt registered [12454.608797] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12455.098449] Key type .llcrypt unregistered [12455.101929] Key type ._llcrypt unregistered [12487.144667] Key type ._llcrypt registered [12487.147132] Key type .llcrypt registered [12487.280509] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12487.635463] Key type .llcrypt unregistered [12487.638085] Key type ._llcrypt unregistered [12522.860316] Key type ._llcrypt registered [12522.863380] Key type .llcrypt registered [12523.031411] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12523.407257] Key type .llcrypt unregistered [12523.410192] Key type ._llcrypt unregistered [12554.566989] Key type ._llcrypt registered [12554.569579] Key type .llcrypt registered [12554.691908] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12555.095331] Key type .llcrypt unregistered [12555.101322] Key type ._llcrypt unregistered [12586.232271] Key type ._llcrypt registered [12586.234781] Key type .llcrypt registered [12586.348345] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12586.613650] Key type .llcrypt unregistered [12586.615855] Key type ._llcrypt unregistered [12617.665260] Key type ._llcrypt registered [12617.669459] Key type .llcrypt registered [12617.789741] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12618.132315] Key type .llcrypt unregistered [12618.134193] Key type ._llcrypt unregistered [12685.190064] Key type ._llcrypt registered [12685.198649] Key type .llcrypt registered [12685.408530] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12685.713319] Key type .llcrypt unregistered [12685.716450] Key type ._llcrypt unregistered [12715.574290] Key type ._llcrypt registered [12715.577341] Key type .llcrypt registered [12715.670241] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12716.021478] Key type .llcrypt unregistered [12716.024837] Key type ._llcrypt unregistered [12750.043988] Key type ._llcrypt registered [12750.051145] Key type .llcrypt registered [12750.248980] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [12750.609126] Key type .llcrypt unregistered [12750.621539] Key type ._llcrypt unregistered [12812.756571] Key type ._llcrypt registered [12812.760139] Key type .llcrypt registered [12812.940849] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [12829.137444] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [12830.907031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12831.095219] alg: No test for adler32 (adler32-zlib) [12832.312806] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [12832.643473] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [12834.391220] Key type lgssc registered [12835.770242] Lustre: Echo OBD driver; http://www.lustre.org/ [12880.419649] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12880.458912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12882.022728] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12882.158735] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12882.340708] Lustre: tst250fs-MDT0000: new disk, initializing [12882.461812] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12882.505052] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12887.838310] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12897.940490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12898.051129] Lustre: 332412:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12898.105969] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12898.109952] Lustre: Skipped 1 previous similar message [12898.164492] Lustre: tst250fs-MDT0001: new disk, initializing [12898.294635] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12898.342860] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12898.355761] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12902.828654] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12907.942026] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12915.296791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12915.576528] Lustre: tst250fs-OST0000: new disk, initializing [12915.579611] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12915.657659] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12916.392554] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12916.407982] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12916.550434] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12922.409557] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12934.524356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12934.795928] Lustre: tst250fs-OST0001: new disk, initializing [12934.800406] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12934.893575] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12940.993610] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12941.016409] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12941.194218] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12942.957654] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [12949.453289] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12951.460983] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [12951.465913] Lustre: Skipped 1 previous similar message [12972.032879] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12972.040392] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12972.049371] Lustre: Skipped 3 previous similar messages [12972.067384] Lustre: Skipped 3 previous similar messages [12977.121843] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12977.133369] Lustre: Skipped 2 previous similar messages [12982.245512] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12982.260484] Lustre: Skipped 3 previous similar messages [12986.335222] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12986.702483] Lustre: server umount tst250fs-MDT0000 complete [12990.540454] LustreError: 332406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788610987 with bad export cookie 10946441706937157755 [12990.556324] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12991.031650] Lustre: server umount tst250fs-MDT0001 complete [13004.839921] Lustre: server umount tst250fs-OST0000 complete [13007.840166] Lustre: 330636:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788610989/real 1788610989] req@ffff9f467e350a80 x1875494389750144/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788611005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13007.862168] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13009.115765] Lustre: server umount tst250fs-OST0001 complete [13029.444578] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [13037.727607] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [13071.435714] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 08:24:27 (1788611067) [13073.017146] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13089.534949] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [13100.872983] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [13137.402144] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12831 sec ========= 08:25:33 (1788611133) [13139.738424] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:25:35 (1788611135) === [13143.877033] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:25:39 (1788611139) === [13190.518720] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [13193.401264] Key type lgssc unregistered [13193.681946] LNet: 343895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13193.706390] LNetError: 343895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13193.735556] LNet: Removed LNI 192.168.201.134@tcp [13194.854271] Key type .llcrypt unregistered [13194.858092] Key type ._llcrypt unregistered