[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 538026680 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003299] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008693] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.012143] LSM: Security Framework initializing [ 0.014045] Yama: becoming mindful. [ 0.015037] SELinux: Initializing. [ 0.016087] *** VALIDATE selinux *** [ 0.024576] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029435] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032133] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033138] *** VALIDATE tmpfs *** [ 0.034473] *** VALIDATE proc *** [ 0.036231] *** VALIDATE cgroup *** [ 0.037011] *** VALIDATE cgroup2 *** [ 0.038273] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039174] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041043] Spectre V2 : User space: Vulnerable [ 0.042013] Speculative Store Bypass: Vulnerable [ 0.045862] debug: unmapping init [mem 0xffffffffbdc59000-0xffffffffbdc60fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048677] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049027] ... version: 2 [ 0.050018] ... bit width: 48 [ 0.051014] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053014] ... max period: 00007fffffffffff [ 0.054015] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.056370] rcu: Hierarchical SRCU implementation. [ 0.058535] smp: Bringing up secondary CPUs ... [ 0.059652] x86: Booting SMP configuration: [ 0.060029] .... node #0, CPUs: #1 #2 #3 [ 0.063231] smp: Brought up 1 node, 4 CPUs [ 0.065015] smpboot: Max logical packages: 1 [ 0.066026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150021] node 0 deferred pages initialised in 82ms [ 0.152460] devtmpfs: initialized [ 0.153180] x86/mm: Memory block size: 128MB [ 0.156011] gcov: version magic: 0x41383552 [ 0.158298] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163275] pinctrl core: initialized pinctrl subsystem [ 0.165113] [ 0.165508] ************************************************************* [ 0.166007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168010] ** ** [ 0.169007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170007] ** ** [ 0.172008] ** This means that this kernel is built to expose internal ** [ 0.173007] ** IOMMU data structures, which may compromise security on ** [ 0.175011] ** your system. ** [ 0.176006] ** ** [ 0.177007] ** If you see this message and you are not debugging the ** [ 0.178007] ** kernel, report this immediately to your vendor! ** [ 0.179006] ** ** [ 0.181006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182007] ************************************************************* [ 0.183569] NET: Registered protocol family 16 [ 0.185342] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.187045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.188034] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.190547] cpuidle: using governor menu [ 0.192552] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.195588] PCI: Using configuration type 1 for base access [ 0.197131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.206166] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.207029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.209076] cryptd: max_cpu_qlen set to 1000 [ 0.212287] ACPI: Added _OSI(Module Device) [ 0.214013] ACPI: Added _OSI(Processor Device) [ 0.215009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.216010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.221329] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.227579] ACPI: Interpreter enabled [ 0.229046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230011] ACPI: Using IOAPIC for interrupt routing [ 0.231098] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.233237] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239751] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.242039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.245020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.248085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.250891] acpiphp: Slot [2] registered [ 0.252058] acpiphp: Slot [5] registered [ 0.252881] acpiphp: Slot [6] registered [ 0.253050] acpiphp: Slot [7] registered [ 0.253966] acpiphp: Slot [8] registered [ 0.255064] acpiphp: Slot [9] registered [ 0.255773] acpiphp: Slot [10] registered [ 0.256059] acpiphp: Slot [3] registered [ 0.256890] acpiphp: Slot [4] registered [ 0.258054] acpiphp: Slot [11] registered [ 0.258869] acpiphp: Slot [12] registered [ 0.260058] acpiphp: Slot [13] registered [ 0.260814] acpiphp: Slot [14] registered [ 0.261050] acpiphp: Slot [15] registered [ 0.261932] acpiphp: Slot [16] registered [ 0.263061] acpiphp: Slot [17] registered [ 0.263931] acpiphp: Slot [18] registered [ 0.264053] acpiphp: Slot [19] registered [ 0.264849] acpiphp: Slot [20] registered [ 0.266069] acpiphp: Slot [21] registered [ 0.266947] acpiphp: Slot [22] registered [ 0.268065] acpiphp: Slot [23] registered [ 0.268906] acpiphp: Slot [24] registered [ 0.269050] acpiphp: Slot [25] registered [ 0.269857] acpiphp: Slot [26] registered [ 0.271049] acpiphp: Slot [27] registered [ 0.271785] acpiphp: Slot [28] registered [ 0.273076] acpiphp: Slot [29] registered [ 0.274038] acpiphp: Slot [30] registered [ 0.274862] acpiphp: Slot [31] registered [ 0.275037] PCI host bridge to bus 0000:00 [ 0.275754] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.277015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.278012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.280024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.283023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.285024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.287185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.291161] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.293540] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.303751] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.310061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.313022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.316021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.319027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.322632] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.325824] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.328049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.332645] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.336017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.348014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.353021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.357000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.376024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.384028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.400023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.411923] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.424021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.433024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.456018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.471282] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.479019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.489020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.502018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.517925] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.534020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.543018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.579020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.601315] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.613019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.624018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.649022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.662073] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.680017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.689033] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.718022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.728725] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.730406] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.732315] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.733224] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.735127] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.740044] iommu: Default domain type: Passthrough [ 0.741364] SCSI subsystem initialized [ 0.742159] ACPI: bus type USB registered [ 0.744125] usbcore: registered new interface driver usbfs [ 0.746080] usbcore: registered new interface driver hub [ 0.747077] usbcore: registered new device driver usb [ 0.749196] pps_core: LinuxPPS API ver. 1 registered [ 0.751014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.754073] PTP clock support registered [ 0.756149] EDAC MC: Ver: 3.0.0 [ 0.758149] PCI: Using ACPI for IRQ routing [ 0.760868] NetLabel: Initializing [ 0.761011] NetLabel: domain hash size = 128 [ 0.762010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.764084] NetLabel: unlabeled traffic allowed by default [ 0.766178] vgaarb: loaded [ 0.767235] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.769012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.777329] clocksource: Switched to clocksource kvm-clock [ 0.892851] VFS: Disk quotas dquot_6.6.0 [ 0.895067] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.898488] *** VALIDATE ramfs *** [ 0.899642] *** VALIDATE hugetlbfs *** [ 0.901371] pnp: PnP ACPI init [ 0.904119] pnp: PnP ACPI: found 6 devices [ 0.926121] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.929230] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.931073] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.933051] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.935276] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.937324] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.940216] NET: Registered protocol family 2 [ 0.942634] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.947138] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.950820] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.957054] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.960897] TCP: Hash tables configured (established 65536 bind 65536) [ 0.963708] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.966803] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.969640] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.972610] NET: Registered protocol family 1 [ 0.976225] RPC: Registered named UNIX socket transport module. [ 0.979191] RPC: Registered udp transport module. [ 0.981616] RPC: Registered tcp transport module. [ 0.983983] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.987302] NET: Registered protocol family 44 [ 0.988604] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.990252] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.991856] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.994317] PCI: CLS 0 bytes, default 64 [ 0.996161] Unpacking initramfs... [ 2.327622] debug: unmapping init [mem 0xffff9811fcc54000-0xffff9811fffbffff] [ 2.331401] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.333300] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.335666] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.815287] Initialise system trusted keyrings [ 2.817836] Key type blacklist registered [ 2.820539] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.831087] zbud: loaded [ 2.834224] *** VALIDATE nfs *** [ 2.835952] *** VALIDATE nfs4 *** [ 2.838040] pstore: using deflate compression [ 2.841856] Platform Keyring initialized [ 2.932841] NET: Registered protocol family 38 [ 2.934418] Key type asymmetric registered [ 2.935903] Asymmetric key parser 'x509' registered [ 2.938252] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.942608] io scheduler mq-deadline registered [ 2.944392] io scheduler kyber registered [ 2.946971] io scheduler bfq registered [ 2.949640] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.953877] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.958212] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.962238] ACPI: Power Button [PWRF] [ 2.968339] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.974182] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.987836] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.998166] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.041441] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.070764] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.097798] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.102532] Non-volatile memory driver v1.3 [ 3.104083] Linux agpgart interface v0.103 [ 3.136397] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.138650] vda: detected capacity change from 0 to 74825728 [ 3.151749] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.154086] vdb: detected capacity change from 0 to 1073741824 [ 3.165383] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.168160] vdc: detected capacity change from 0 to 2621440000 [ 3.180916] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.183407] vdd: detected capacity change from 0 to 2621440000 [ 3.200456] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.203455] vde: detected capacity change from 0 to 4294967296 [ 3.220084] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.222514] vdf: detected capacity change from 0 to 4294967296 [ 3.229938] libphy: Fixed MDIO Bus: probed [ 3.235900] usbcore: registered new interface driver usbserial_generic [ 3.238597] usbserial: USB Serial support registered for generic [ 3.240922] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.246152] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.247897] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.250191] mousedev: PS/2 mouse device common for all mice [ 3.253591] rtc_cmos 00:05: RTC can wake from S4 [ 3.256479] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.257951] rtc_cmos 00:05: registered as rtc0 [ 3.262511] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.264508] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.266909] intel_pstate: CPU model not supported [ 3.271259] hid: raw HID events driver (C) Jiri Kosina [ 3.274380] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.275192] usbcore: registered new interface driver usbhid [ 3.279086] usbhid: USB HID core driver [ 3.280969] drop_monitor: Initializing network drop monitor service [ 3.283457] Initializing XFRM netlink socket [ 3.285879] NET: Registered protocol family 10 [ 3.288847] Segment Routing with IPv6 [ 3.290391] NET: Registered protocol family 17 [ 3.292743] mpls_gso: MPLS GSO support [ 3.297646] RAS: Correctable Errors collector initialized. [ 3.299381] AVX version of gcm_enc/dec engaged. [ 3.302215] AES CTR mode by8 optimization enabled [ 3.376548] sched_clock: Marking stable (3376518318, 0)->(4292132869, -915614551) [ 3.380130] registered taskstats version 1 [ 3.382154] Loading compiled-in X.509 certificates [ 3.383611] zswap: loaded using pool lzo/zbud [ 3.407806] Key type big_key registered [ 3.421210] Key type encrypted registered [ 3.422801] ima: No TPM chip found, activating TPM-bypass! [ 3.424899] ima: Allocated hash algorithm: sha1 [ 3.426576] ima: No architecture policies found [ 3.428401] evm: Initialising EVM extended attributes: [ 3.430287] evm: security.selinux [ 3.431529] evm: security.ima [ 3.432629] evm: security.capability [ 3.433865] evm: HMAC attrs: 0x1 [ 3.436267] rtc_cmos 00:05: setting system clock to 2026-08-23 03:02:24 UTC (1787454144) [ 3.442540] debug: unmapping init [mem 0xffffffffbec03000-0xffffffffbedfffff] [ 3.445573] debug: unmapping init [mem 0xffffffffbd982000-0xffffffffbdc58fff] [ 3.459096] Write protecting the kernel read-only data: 28672k [ 3.462394] debug: unmapping init [mem 0xffffffffbc003000-0xffffffffbc1fffff] [ 3.465269] debug: unmapping init [mem 0xffffffffbc914000-0xffffffffbc9fffff] [ 3.500651] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.508297] systemd[1]: Detected virtualization kvm. [ 3.509952] systemd[1]: Detected architecture x86-64. [ 3.511800] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.562963] systemd[1]: No hostname configured. [ 3.564211] systemd[1]: Set hostname to . [ 3.569537] random: systemd: uninitialized urandom read (16 bytes read) [ 3.571435] systemd[1]: Initializing machine ID from random generator. [ 3.631575] random: ln: uninitialized urandom read (6 bytes read) [ 3.711352] random: systemd: uninitialized urandom read (16 bytes read) [ 3.713934] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.718060] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.721148] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.323183] device-mapper: uevent: version 1.0.3 [ 4.325753] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.143809] virtio_net virtio0 ens2: renamed from eth0 [ 5.161243] random: fast init done [ 5.168961] scsi host0: ata_piix [ 5.209508] scsi host1: ata_piix [ 5.210746] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.212548] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.948437] random: crng init done [ 9.950204] random: 7 urandom warning(s) missed due to ratelimiting [ 10.747289] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.173530] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.771005] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.638870] SELinux: Disabled at runtime. [ 16.771469] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.794610] systemd[1]: Detected virtualization kvm. [ 16.798090] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.909920] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.923285] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.952607] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.967817] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.979807] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.006443] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.019885] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ 19.653257] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 20.864821] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.931396] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.100745] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.426323] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.491702] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 27.464189] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 28.263933] NFS: Registering the id_resolver key type [ 28.279054] Key type id_resolver registered [ 28.283212] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Login Service. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg246-server login: [ 53.903100] hrtimer: interrupt took 4845558 ns [ 88.054378] libcfs: loading out-of-tree module taints kernel. [ 88.107376] Key type ._llcrypt registered [ 88.109510] Key type .llcrypt registered [ 88.196554] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 108.534242] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 110.154625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 110.170066] alg: No test for adler32 (adler32-zlib) [ 111.636638] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 112.483430] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 114.263966] Key type lgssc registered [ 115.853438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 133.132446] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 177.581868] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 191.708580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 191.736832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.030707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.073380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.192165] Lustre: lustre-MDT0000: new disk, initializing [ 193.286587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.302860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 197.805646] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.011444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 212.132416] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 212.184989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 212.193601] Lustre: Skipped 1 previous similar message [ 212.279573] Lustre: lustre-MDT0001: new disk, initializing [ 212.349992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 212.376430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 212.389017] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.806790] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.296410] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 230.441921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 230.736653] Lustre: lustre-OST0000: new disk, initializing [ 230.744834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 230.752301] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 230.761594] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 230.823269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.071439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 236.074776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 236.105983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 236.166152] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.749979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.941682] Lustre: lustre-OST0001: new disk, initializing [ 249.959775] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 249.965721] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 250.045055] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 255.519519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 255.531184] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 255.574921] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 255.844392] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 267.618524] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 276.109392] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 282.742816] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing check_logdir /tmp/testlogs/ [ 289.829644] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing yml_node [ 297.258966] Lustre: DEBUG MARKER: Client: 2.17.57.63 [ 300.860508] Lustre: DEBUG MARKER: MDS: 2.17.57.63 [ 303.872075] Lustre: DEBUG MARKER: OSS: 2.17.57.63 [ 305.818342] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 23:07:26 EDT 2026 [ 325.304311] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 327.031750] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 345.569646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 345.575595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.589718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.619238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.623450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.636937] Lustre: Skipped 2 previous similar messages [ 347.650439] Lustre: Skipped 2 previous similar messages [ 352.736951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.747695] Lustre: Skipped 4 previous similar messages [ 357.343925] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 357.512628] Lustre: server umount lustre-MDT0000 complete [ 360.915584] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787454501 with bad export cookie 13760007294970710730 [ 360.919513] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 360.940466] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 362.980907] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.993201] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.993276] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 362.995311] Lustre: Skipped 1 previous similar message [ 363.007838] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 365.983895] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.999862] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 367.324708] Lustre: server umount lustre-MDT0001 complete [ 370.946849] Lustre: server umount lustre-OST0000 complete [ 374.473877] Lustre: server umount lustre-OST0001 complete [ 380.933923] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 388.750894] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 425.343279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.645569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 425.690230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 425.831953] Lustre: lustre-MDT0000: new disk, initializing [ 425.939934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.952149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 430.972288] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 442.487649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.699231] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 442.804132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 442.813606] Lustre: Skipped 1 previous similar message [ 442.885439] Lustre: lustre-MDT0001: new disk, initializing [ 442.980979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 443.025043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 443.038500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 448.879887] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 459.668068] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 465.133409] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 471.326519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 471.631290] Lustre: lustre-OST0000: new disk, initializing [ 471.634156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 471.640475] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 471.752597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.853738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 472.872682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 473.036737] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 478.146141] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 486.868572] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 490.919606] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 491.126280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 495.177946] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 495.340798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 498.658435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.663446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 498.670000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 498.670024] Lustre: Skipped 3 previous similar messages [ 498.670847] Lustre: Skipped 1 previous similar message [ 503.354464] Lustre: server umount lustre-OST0000 complete [ 508.901106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.017361] LustreError: 15943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.054606] LustreError: 15943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 514.273269] Lustre: server umount lustre-MDT0000 complete [ 518.113309] LustreError: 17052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.127599] LustreError: 17052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 518.600264] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787454659 with bad export cookie 13760007294970712263 [ 518.601806] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.611885] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 519.108157] Lustre: server umount lustre-MDT0001 complete [ 525.327671] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 23:11:05 (1787454665) [ 549.051307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.482203] LustreError: 19919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.558579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.044730] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 559.089873] LustreError: 19919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.113061] LustreError: 19919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 563.324716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.265321] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 578.993907] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 587.715295] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 590.791866] Lustre: Failing over lustre-MDT0000 [ 591.491469] Lustre: server umount lustre-MDT0000 complete [ 593.377902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.391609] Lustre: Skipped 1 previous similar message [ 593.401969] LustreError: 19941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.417854] LustreError: 19941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 609.639629] Lustre: 3647:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787454734/real 1787454734] req@ffff981143781f80 x1874281630956288/t0(0) o400->MGC192.168.202.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787454750 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 609.697645] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.777343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.318956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 619.328823] Lustre: Skipped 1 previous similar message [ 619.422817] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 624.627424] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 624.640278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 624.681147] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 625.274504] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 635.262598] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 639.970448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 639.987490] LustreError: Skipped 1 previous similar message [ 639.995152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 640.005591] Lustre: Skipped 1 previous similar message [ 640.015622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 640.020188] Lustre: Skipped 3 previous similar messages [ 645.837462] Lustre: server umount lustre-MDT0000 complete [ 649.185332] LustreError: 19914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 649.198960] LustreError: 19914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 649.370926] LustreError: 19902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787454790 with bad export cookie 13760007294970713677 [ 649.376733] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 649.386603] LustreError: 19902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 649.705991] Lustre: server umount lustre-MDT0001 complete [ 658.263743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.856336] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 673.346809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 677.986583] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 685.807786] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 691.719979] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 694.373664] Lustre: Failing over lustre-MDT0000 [ 694.779903] Lustre: server umount lustre-MDT0000 complete [ 698.337338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 698.350950] Lustre: Skipped 2 previous similar messages [ 714.719515] Lustre: 3646:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787454839/real 1787454839] req@ffff9811430a9180 x1874281630986112/t0(0) o400->MGC192.168.202.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787454855 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 714.768936] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 714.800571] LustreError: 23259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 714.824271] LustreError: 23259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 715.455853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 724.511848] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 729.473140] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 729.569250] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 729.573938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 729.590297] Lustre: Skipped 1 previous similar message [ 729.613881] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 740.912296] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 750.054695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 750.066765] Lustre: Skipped 2 previous similar messages [ 750.076914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 751.879511] Lustre: server umount lustre-MDT0000 complete [ 755.966560] LustreError: 23240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787454897 with bad export cookie 13760007294970714615 [ 755.981836] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 755.984529] LustreError: 23240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 756.387954] Lustre: server umount lustre-MDT0001 complete [ 766.391583] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 23:15:06 (1787454906) [ 783.035614] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 793.628776] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 835.588728] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 846.854248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.135205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 847.174482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 847.262632] Lustre: lustre-MDT0000: new disk, initializing [ 847.333194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 847.337477] Lustre: Skipped 3 previous similar messages [ 847.350605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 851.902904] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 863.754405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.941248] Lustre: 29635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 864.277924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 868.804962] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 878.363895] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 884.580830] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 891.779505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 892.264898] Lustre: 31605:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 894.170551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 894.248074] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 894.355163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 894.388886] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 898.939652] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 908.790669] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 914.696042] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 935.399361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.400369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 935.428436] Lustre: Skipped 2 previous similar messages [ 935.430856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 940.525585] Lustre: server umount lustre-OST0000 complete [ 950.444325] Lustre: server umount lustre-MDT0000 complete [ 950.752718] LustreError: 30524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 950.765338] LustreError: 30524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 954.211203] LustreError: 29628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787455095 with bad export cookie 13760007294970715021 [ 954.212919] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.220473] LustreError: 29628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 954.528594] Lustre: server umount lustre-MDT0001 complete [ 966.986158] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 971.170125] Key type lgssc unregistered [ 971.454309] LNet: 33585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.463065] LNetError: 33585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.478476] LNet: Removed LNI 192.168.202.146@tcp [ 972.472175] Key type .llcrypt unregistered [ 972.474376] Key type ._llcrypt unregistered [ 999.717342] Key type ._llcrypt registered [ 999.720763] Key type .llcrypt registered [ 999.863854] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1001.184063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1001.206710] alg: No test for adler32 (adler32-zlib) [ 1002.281387] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1002.576389] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1004.295194] Key type lgssc registered [ 1005.522671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1013.284965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1013.316473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.000329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1020.060817] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1025.508659] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1025.540942] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1029.027073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.411629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1034.857247] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1042.627730] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1048.432783] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1055.456558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1055.847566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1059.979646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1062.320810] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1071.413615] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1074.295919] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1091.392432] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 23:20:31 (1787455231) [ 1093.770368] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1095.819145] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1104.126362] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 23:20:44 (1787455244) [ 1115.289338] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 23:20:55 (1787455255) [ 1116.562914] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1125.470967] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1125.478462] Lustre: Skipped 2 previous similar messages [ 1127.711628] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 23:21:07 (1787455267) [ 1129.846616] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1129.854668] Lustre: Skipped 2 previous similar messages [ 1139.110473] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1139.116329] Lustre: Skipped 4 previous similar messages [ 1154.037867] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 23:21:33 (1787455293) [ 1157.170422] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1160.249494] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1170.068901] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 23:21:50 (1787455310) [ 1172.254969] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1172.258433] Lustre: Skipped 6 previous similar messages [ 1188.733529] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 23:22:08 (1787455328) [ 1207.776124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1207.782019] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.796792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1207.834840] Lustre: Skipped 1 previous similar message [ 1212.898944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1212.917858] Lustre: Skipped 2 previous similar messages [ 1214.127638] Lustre: server umount lustre-OST0000 complete [ 1219.040465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1219.051166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.059536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1223.144045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1223.157901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.017221] Lustre: server umount lustre-MDT0000 complete [ 1228.262403] LustreError: 35007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1228.314944] LustreError: 35007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1229.030147] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787455370 with bad export cookie 8060569195054534592 [ 1229.034541] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1229.053059] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1229.740468] Lustre: server umount lustre-MDT0001 complete [ 1242.451244] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 1245.472411] Key type lgssc unregistered [ 1245.771636] LNet: 42864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1245.783068] LNetError: 42864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1245.807647] LNet: Removed LNI 192.168.202.146@tcp [ 1246.818344] Key type .llcrypt unregistered [ 1246.819960] Key type ._llcrypt unregistered [ 1265.620063] Key type ._llcrypt registered [ 1265.624220] Key type .llcrypt registered [ 1265.782751] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 1283.484545] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1285.055146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1285.074206] alg: No test for adler32 (adler32-zlib) [ 1286.150211] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1286.368068] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1288.103245] Key type lgssc registered [ 1289.247693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1328.276643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1328.296136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1329.774419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1329.896854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1330.011299] Lustre: lustre-MDT0000: new disk, initializing [ 1330.143172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1330.177593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1337.076666] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1348.643538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.755699] Lustre: 46019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1348.792775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1348.796124] Lustre: Skipped 1 previous similar message [ 1348.897990] Lustre: lustre-MDT0001: new disk, initializing [ 1349.056145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1349.103085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1349.112729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1354.677532] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1365.868066] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1373.585996] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1380.525325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1380.830655] Lustre: lustre-OST0000: new disk, initializing [ 1380.840899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1380.859788] Lustre: 47990:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1380.985860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.100839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1387.112046] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1387.281331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1387.411514] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1396.873243] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1401.381649] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1401.596939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1407.030974] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1407.297769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1413.094957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1413.098297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.107390] LustreError: Skipped 1 previous similar message [ 1413.107571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1413.148208] Lustre: Skipped 1 previous similar message [ 1415.547344] Lustre: server umount lustre-OST0000 complete [ 1420.771105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1420.789388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.814545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1420.826372] Lustre: Skipped 1 previous similar message [ 1423.333749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.335191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1423.347928] Lustre: Skipped 1 previous similar message [ 1426.813398] Lustre: server umount lustre-MDT0000 complete [ 1431.547967] LustreError: 46009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787455572 with bad export cookie 11172987241217478810 [ 1431.560414] LustreError: 46009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1431.565723] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1431.913732] Lustre: server umount lustre-MDT0001 complete [ 1445.381369] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 23:26:24 (1787455584) [ 1452.418870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.748623] LustreError: 50450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.762588] LustreError: 50450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1452.801204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1457.681810] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1458.152602] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1462.246578] LustreError: 50450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1466.784166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.264484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1472.086293] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1479.252874] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1485.883511] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1494.104431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1494.557632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1502.170543] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1511.058308] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1515.857179] LustreError: 52978:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1517.008057] Lustre: Modifying parameter general.timeout=1 in log params [ 1518.214906] Lustre: Modifying parameter general.timeout=2 in log params [ 1521.156430] Lustre: Modifying parameter general.timeout=5 in log params [ 1521.168586] Lustre: Skipped 2 previous similar messages [ 1525.946949] Lustre: Modifying parameter general.timeout=10 in log params [ 1525.955951] Lustre: Skipped 4 previous similar messages [ 1534.577554] Lustre: Modifying parameter general.timeout=18 in log params [ 1534.580182] Lustre: Skipped 7 previous similar messages [ 1550.966270] Lustre: Modifying parameter general.timeout=37 in log params [ 1550.969055] Lustre: Skipped 18 previous similar messages [ 1583.243895] Lustre: Modifying parameter general.timeout=69 in log params [ 1583.248952] Lustre: Skipped 31 previous similar messages [ 1650.725665] Lustre: Disabling parameter general.timeout= in log params [ 1650.735768] Lustre: Skipped 60 previous similar messages [ 1654.154479] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 23:29:54 (1787455794) [ 1657.399582] Lustre: 59930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1659.460470] Lustre: 60028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1663.302640] Lustre: 60226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1663.321545] Lustre: 60226:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1695.975462] Lustre: 61798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1695.994965] Lustre: 61798:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1696.056986] LustreError: 61798:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1702.571665] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 23:30:42 (1787455842) [ 1716.194960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1716.200888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1716.217710] Lustre: Skipped 1 previous similar message [ 1716.232727] Lustre: Skipped 2 previous similar messages [ 1721.316691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1721.323402] Lustre: Skipped 2 previous similar messages [ 1721.915584] Lustre: server umount lustre-MDT0000 complete [ 1726.434853] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1726.476182] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1726.591073] LustreError: 50429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787455867 with bad export cookie 11172987241217479692 [ 1726.600548] LustreError: 50429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1726.603919] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1726.979460] Lustre: server umount lustre-MDT0001 complete [ 1741.464933] Lustre: server umount lustre-OST0000 complete [ 1770.028759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.319584] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1770.579114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.994624] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1784.254694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.376679] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1784.425329] Lustre: 64328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1790.318235] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1797.760515] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1804.793288] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1811.912641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1812.093326] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1812.339230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1812.345840] Lustre: Skipped 1 previous similar message [ 1820.996235] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1830.336495] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1832.025813] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1832.035108] Lustre: Skipped 23 previous similar messages [ 1832.076907] Lustre: 66896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1841.121558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1841.130968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.145805] Lustre: Skipped 1 previous similar message [ 1841.161140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1844.474559] Lustre: server umount lustre-OST0000 complete [ 1851.364972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1851.373144] LustreError: Skipped 1 previous similar message [ 1851.376365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.388943] Lustre: Skipped 1 previous similar message [ 1854.579332] Lustre: server umount lustre-MDT0000 complete [ 1856.996480] LustreError: 64342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1857.025478] LustreError: 64342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1858.341046] LustreError: 64321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787455999 with bad export cookie 11172987241217482716 [ 1858.347279] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1858.350464] LustreError: 64321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1858.696539] Lustre: server umount lustre-MDT0001 complete [ 1871.181769] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 1874.289707] Key type lgssc unregistered [ 1874.725738] LNet: 68106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1874.733319] LNetError: 68106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.755824] LNet: Removed LNI 192.168.202.146@tcp [ 1875.770435] Key type .llcrypt unregistered [ 1875.774865] Key type ._llcrypt unregistered [ 1905.366221] Key type ._llcrypt registered [ 1905.370414] Key type .llcrypt registered [ 1905.553813] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1906.588316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1906.656402] alg: No test for adler32 (adler32-zlib) [ 1907.782616] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1907.960561] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1909.679197] Key type lgssc registered [ 1910.893618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1919.407818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1919.444914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.165565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1925.560286] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1931.238818] LustreError: 69525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1931.267751] LustreError: 69525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1934.789466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.469568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1941.441944] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1949.538885] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1956.684641] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1964.093711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1964.642452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.474576] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1980.401355] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1987.752774] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2000.876027] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.888628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2000.911760] Lustre: Skipped 1 previous similar message [ 2003.671538] Lustre: server umount lustre-MDT0000 complete [ 2006.002820] LustreError: 69824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2006.015378] LustreError: 69824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2008.167114] LustreError: 69506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787456149 with bad export cookie 14041367326704976832 [ 2008.176969] LustreError: 69506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2008.177744] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2008.485792] Lustre: server umount lustre-MDT0001 complete [ 2022.468533] Lustre: server umount lustre-OST0000 complete [ 2056.204614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2056.353508] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2056.369338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2056.522296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2062.960473] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2072.766339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.019948] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2073.057578] Lustre: 74396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2073.890930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2079.278327] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2088.350312] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2095.063661] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2102.499311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2102.850534] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2103.237590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2110.104248] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2119.172658] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2121.204762] Lustre: 76975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2123.188346] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2129.381895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.398346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2129.400174] Lustre: Skipped 1 previous similar message [ 2129.406370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2129.406379] Lustre: Skipped 2 previous similar messages [ 2134.518627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2134.530176] Lustre: Skipped 1 previous similar message [ 2135.241374] Lustre: server umount lustre-OST0000 complete [ 2139.617705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.639945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.653987] Lustre: Skipped 2 previous similar messages [ 2144.738461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2144.741615] Lustre: Skipped 2 previous similar messages [ 2145.266438] Lustre: server umount lustre-MDT0000 complete [ 2149.862800] LustreError: 74409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.891277] LustreError: 74409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2149.907357] LustreError: 75144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787456290 with bad export cookie 14041367326704977665 [ 2149.916717] LustreError: 75144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2149.919569] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2150.294979] Lustre: server umount lustre-MDT0001 complete [ 2164.590983] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2167.775514] Key type lgssc unregistered [ 2168.050386] LNet: 78189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.055634] LNetError: 78189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.074076] LNet: Removed LNI 192.168.202.146@tcp [ 2169.143759] Key type .llcrypt unregistered [ 2169.148769] Key type ._llcrypt unregistered [ 2196.190249] Key type ._llcrypt registered [ 2196.195360] Key type .llcrypt registered [ 2196.266780] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2197.210631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2197.380028] alg: No test for adler32 (adler32-zlib) [ 2198.463443] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2198.739824] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2200.458107] Key type lgssc registered [ 2201.795190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2209.987824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2210.002873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.602109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.697953] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2221.028442] LustreError: 79603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2221.057613] LustreError: 79603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2226.147443] LustreError: 79602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.591604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.912426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2235.154278] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2245.877991] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2253.624216] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2260.670034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2261.155102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2267.330394] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2276.343195] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2288.644594] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2289.531983] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2290.755939] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2292.917332] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2292.924815] Lustre: Skipped 1 previous similar message [ 2297.461867] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2297.469049] Lustre: Skipped 3 previous similar messages [ 2306.486959] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2306.497205] Lustre: Skipped 8 previous similar messages [ 2323.199651] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2323.209737] Lustre: Skipped 15 previous similar messages [ 2355.708766] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2355.713653] Lustre: Skipped 31 previous similar messages [ 2399.159250] Lustre: 87730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2402.365129] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 23:42:22 (1787456542) [ 2404.558235] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2406.456320] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 23:42:26 (1787456546) [ 2413.024979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2413.036955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.047967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.085349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.099242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.108094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.136250] Lustre: Skipped 1 previous similar message [ 2420.199056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2425.314177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2425.324610] Lustre: Skipped 2 previous similar messages [ 2426.338360] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2426.549877] Lustre: server umount lustre-OST0000 complete [ 2434.538531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2434.549866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.578362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2437.444763] Lustre: server umount lustre-MDT0000 complete [ 2440.674938] LustreError: 79603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.728834] LustreError: 79603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2441.727124] LustreError: 79582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787456582 with bad export cookie 9345966004407074094 [ 2441.730995] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2441.744167] LustreError: 79582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2442.342936] Lustre: server umount lustre-MDT0001 complete [ 2454.085094] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2456.794157] Key type lgssc unregistered [ 2457.029752] LNet: 88987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2457.035927] LNetError: 88987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2457.057814] LNet: Removed LNI 192.168.202.146@tcp [ 2458.023817] Key type .llcrypt unregistered [ 2458.029190] Key type ._llcrypt unregistered [ 2463.423630] Key type ._llcrypt registered [ 2463.430344] Key type .llcrypt registered [ 2463.646622] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2469.857464] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules [ 2471.566278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2471.591981] alg: No test for adler32 (adler32-zlib) [ 2472.729742] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2473.044582] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2474.750496] Key type lgssc registered [ 2476.228772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2477.676599] LustreError: 90230:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2497.810791] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2500.732237] LustreError: 90230:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2507.729706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2507.746428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.458762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2514.861179] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2519.012713] LustreError: 91255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.044879] LustreError: 91255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2524.127971] LustreError: 91254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.167098] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 23:44:26 (1787456666) [ 2529.253354] LustreError: 91255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2532.547112] LustreError: 91285:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2532.554832] LustreError: 91285:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2532.560973] LustreError: 91285:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2532.914899] Lustre: server umount lustre-MDT0000 complete [ 2549.538093] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2552.615909] Key type lgssc unregistered [ 2552.926329] LNet: 92781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2552.941883] LNetError: 92781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2552.958527] LNet: Removed LNI 192.168.202.146@tcp [ 2553.874479] Key type .llcrypt unregistered [ 2553.883317] Key type ._llcrypt unregistered [ 2573.242336] Key type ._llcrypt registered [ 2573.254139] Key type .llcrypt registered [ 2573.418865] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2574.429073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2574.528070] alg: No test for adler32 (adler32-zlib) [ 2575.707518] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2575.991436] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2577.783178] Key type lgssc registered [ 2579.281470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2590.729505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2590.751678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2592.240310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.952841] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2602.482589] LustreError: 94001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.513105] LustreError: 94001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2607.584992] LustreError: 94000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.613975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2612.343401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2617.541616] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2624.634385] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2630.780361] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2638.665407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2639.796455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2647.436759] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2656.065660] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2714.915361] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 23:47:34 (1787456854) [ 2717.558694] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2719.752485] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 23:47:39 (1787456859) [ 2729.452200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2729.464431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.489307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2732.919450] Lustre: server umount lustre-MDT0000 complete [ 2735.101274] LustreError: 93997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2735.137262] LustreError: 93997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2737.309644] LustreError: 93983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787456878 with bad export cookie 7974867252554594397 [ 2737.312659] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2737.329423] LustreError: 93983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2737.857710] Lustre: server umount lustre-MDT0001 complete [ 2753.227084] Lustre: server umount lustre-OST0000 complete [ 2769.184771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.906281] LustreError: 98300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.240893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2775.532862] LustreError: 98300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.576761] LustreError: 98300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2776.090497] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2785.779358] LustreError: 98301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2785.833315] LustreError: 98301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2788.251996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2789.643886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2795.737867] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2805.190590] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2811.908503] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2830.556836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2830.875255] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2830.883975] LustreError: 98287:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2830.919149] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2830.924618] LustreError: 100386:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2830.935867] LustreError: 100386:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2830.944336] LustreError: 100386:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2831.032462] Lustre: server umount lustre-OST0000 complete [ 2831.035547] LustreError: 100386:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2843.079540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2843.393760] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2843.409421] LustreError: 98289:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2843.445863] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2843.462677] LustreError: 100852:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2843.470874] LustreError: 100852:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2843.478321] LustreError: 100852:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2843.591742] Lustre: server umount lustre-OST0000 complete [ 2843.596605] LustreError: 100852:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2856.327214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2856.598325] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2856.608287] Lustre: Found index 0 for lustre-OST0000, updating log [ 2856.622443] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2856.770318] Lustre: lustre-OST0000: new disk, initializing [ 2856.779384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2856.793163] Lustre: 101388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2857.197420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.154786] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2871.024248] Lustre: Failing over lustre-OST0000 [ 2871.180480] Lustre: server umount lustre-OST0000 complete [ 2876.899201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.924155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2876.934772] Lustre: Skipped 3 previous similar messages [ 2876.949057] Lustre: Skipped 3 previous similar messages [ 2882.022336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2882.037658] Lustre: Skipped 1 previous similar message [ 2886.254130] Lustre: server umount lustre-MDT0000 complete [ 2887.139719] LustreError: 99700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.148271] Lustre: 93314:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787457012/real 1787457012] req@000000009526dfeb x1874284214596608/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787457028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2887.166075] LustreError: 99700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2887.198564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2889.189309] Lustre: 93311:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787457014/real 1787457014] req@00000000eb82cb02 x1874284214596864/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787457030 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2891.891582] LustreError: 100871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787457032 with bad export cookie 7974867252554597141 [ 2891.897678] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2891.911711] LustreError: 100871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2892.815893] Lustre: server umount lustre-MDT0001 complete [ 2906.315978] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 23:50:45 (1787457045) [ 2930.032911] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2943.724642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.050176] LustreError: 103761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2944.204171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2950.539028] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2961.369020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2969.096587] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2973.049111] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2982.269804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2983.502714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2983.508325] Lustre: Skipped 1 previous similar message [ 2983.530728] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2986.540129] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2989.704548] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2989.707721] Lustre: lustre-OST0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2989.731574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2992.186339] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3003.901483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3004.064662] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3004.164513] Lustre: lustre-OST0001: new disk, initializing [ 3004.167930] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3004.176922] Lustre: 106085:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3010.748713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3010.765622] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3010.940409] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3012.951841] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3024.037601] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3032.156462] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3037.793249] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.46@tcp (stopping) [ 3039.207434] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3039.217416] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.876149] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.46@tcp (stopping) [ 3042.896542] Lustre: Skipped 4 previous similar messages [ 3043.916252] Lustre: server umount lustre-MDT0001 complete [ 3046.919522] LustreError: 103762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.952964] LustreError: 103762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3059.335516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.490207] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3059.494401] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3059.592637] Lustre: 103748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3060.625258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3060.633697] Lustre: Skipped 1 previous similar message [ 3061.265086] LustreError: 107621:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3061.275079] LustreError: 107621:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3061.281684] LustreError: 103753:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3061.410880] LustreError: 107627:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.146@tcp: cfg command failed: rc = -17 [ 3061.417778] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3061.417778] [ 3065.853178] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3065.866084] Lustre: Skipped 1 previous similar message [ 3067.310672] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3080.686261] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 23:53:40 (1787457220) [ 3106.241269] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3129.112593] Lustre: 109738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3156.638454] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3161.061766] Lustre: 110873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3264.992407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3264.996304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.001068] Lustre: Skipped 2 previous similar messages [ 3265.004316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3268.589539] Lustre: server umount lustre-MDT0000 complete [ 3270.643511] LustreError: 103761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.674431] LustreError: 103761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 3274.678488] LustreError: 103742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787457415 with bad export cookie 7974867252554598142 [ 3274.679402] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3274.718435] LustreError: 103742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3275.202543] Lustre: server umount lustre-MDT0001 complete [ 3290.825747] Lustre: server umount lustre-OST0000 complete [ 3290.979215] Lustre: 93312:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787457416/real 1787457416] req@ffff981143590a80 x1874284214879616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787457432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3291.023987] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.040928] Lustre: Skipped 3 previous similar messages [ 3294.945748] Lustre: 93314:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787457420/real 1787457420] req@ffff98127dfed880 x1874284214879872/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787457436 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3295.674261] Lustre: server umount lustre-OST0001 complete [ 3303.879086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3317.245288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3327.035484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3339.373723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3371.167580] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3384.959754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.011599] Lustre: lustre-MDT0000: reset Object Index mappings [ 3385.840598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3389.796729] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3401.196044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3401.253441] Lustre: lustre-MDT0001: reset Object Index mappings [ 3408.083862] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3411.849456] Lustre: 115998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3420.667913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3426.818348] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 3429.236481] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3430.892435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3439.656983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3445.747131] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3445.769761] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3447.246895] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3456.583957] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3461.229152] Lustre: 117878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3521.658564] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:01:01 (1787457661) [ 3548.148596] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.154053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3548.183106] Lustre: Skipped 2 previous similar messages [ 3548.199976] Lustre: Skipped 7 previous similar messages [ 3552.834135] Lustre: server umount lustre-MDT0000 complete [ 3553.255520] LustreError: 114852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3553.278739] LustreError: 114852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 3557.049875] LustreError: 114831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787457698 with bad export cookie 7974867252554645203 [ 3557.055711] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3557.073539] LustreError: 114831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3563.715712] Lustre: server umount lustre-MDT0001 complete [ 3574.651032] Lustre: server umount lustre-OST0000 complete [ 3580.019369] LustreError: 118859:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0xbf:0x0].0x0 (ffff98127c5e7c00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3586.491164] Lustre: server umount lustre-OST0001 complete [ 3595.366643] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 3607.389431] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3658.068942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.572443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3658.667840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3658.768760] Lustre: lustre-MDT0000: new disk, initializing [ 3658.896773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3658.900575] Lustre: Skipped 3 previous similar messages [ 3658.922726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3665.609836] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3674.211939] Lustre: server umount lustre-MDT0000 complete [ 3686.082159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3686.518455] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3686.549936] Lustre: 122170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3686.565548] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3692.024716] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3700.600586] Lustre: server umount lustre-MDT0000 complete [ 3713.980973] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:04:13 (1787457853) [ 3716.235898] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3718.121897] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:04:18 (1787457858) [ 3740.984986] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3753.077261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3759.293433] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3769.128467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3769.327558] Lustre: 124224:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3769.432631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3769.438770] Lustre: Skipped 1 previous similar message [ 3769.546308] Lustre: lustre-MDT0001: new disk, initializing [ 3769.749181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3769.766425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3775.455691] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3780.935684] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3788.486554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3788.779986] Lustre: lustre-OST0000: new disk, initializing [ 3788.782950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3788.789503] Lustre: 125800:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3790.027543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3790.035127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3790.199277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3796.475061] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3809.626060] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3809.892467] Lustre: lustre-OST0001: new disk, initializing [ 3809.910420] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3809.942292] Lustre: 126670:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3811.033052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3811.045130] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3811.233974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3818.950606] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3830.125403] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3834.196783] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4288.132039] Lustre: 133099:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4288.159950] Lustre: 133099:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4288.175723] Lustre: 133099:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4288.202655] Lustre: 133099:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4288.217119] Lustre: 133099:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4288.232845] Lustre: 133099:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4580.027341] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:18:38 (1787458718) [ 4599.279778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4599.304065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4599.330736] Lustre: Skipped 3 previous similar messages [ 4599.334138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4599.336969] Lustre: Skipped 4 previous similar messages [ 4604.167110] Lustre: server umount lustre-MDT0000 complete [ 4604.902136] LustreError: 127481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.936080] LustreError: 127481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4608.355360] LustreError: 124214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787458749 with bad export cookie 7974867252554714601 [ 4608.359094] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4608.380513] LustreError: 124214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4610.016936] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4610.035102] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4610.054708] Lustre: Skipped 4 previous similar messages [ 4610.077840] Lustre: Skipped 4 previous similar messages [ 4615.645695] Lustre: server umount lustre-MDT0001 complete [ 4628.005068] Lustre: server umount lustre-OST0000 complete [ 4641.149985] Lustre: server umount lustre-OST0001 complete [ 4652.814425] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 4669.950898] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4717.701581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4717.880526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4717.905960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4717.956262] Lustre: lustre-MDT0000: new disk, initializing [ 4718.013130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4718.017132] Lustre: Skipped 5 previous similar messages [ 4718.031665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4723.287334] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4738.419053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4738.607181] Lustre: 136179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4738.695777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4738.720741] Lustre: Skipped 1 previous similar message [ 4738.872726] Lustre: lustre-MDT0001: new disk, initializing [ 4739.169866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4739.189132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4744.723895] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4757.064146] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4765.269299] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4775.420030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4775.917892] Lustre: lustre-OST0000: new disk, initializing [ 4775.923754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4775.938344] Lustre: 138147:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4777.616593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4777.635650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4777.862173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4785.704430] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4798.525126] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4811.270708] Lustre: lustre-MDD0000: changelog on [ 4814.522073] Lustre: lustre-MDD0001: changelog on [ 6877.521361] Lustre: lustre-MDD0001: changelog off [ 6879.980338] Lustre: 136186:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6881.067567] Lustre: lustre-MDD0000: changelog off [ 6888.419387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6888.429599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6888.444130] Lustre: Skipped 1 previous similar message [ 6888.460728] Lustre: Skipped 3 previous similar messages [ 6893.540585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6893.546138] Lustre: Skipped 1 previous similar message [ 6894.759930] Lustre: server umount lustre-OST0000 complete [ 6899.683047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6899.691153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6899.711064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6903.785747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6903.789268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6905.506026] Lustre: server umount lustre-MDT0000 complete [ 6908.901047] LustreError: 136186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6908.910991] LustreError: 136186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6909.288683] LustreError: 136171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787461050 with bad export cookie 7974867252555493169 [ 6909.294358] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6909.297879] LustreError: 136171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6909.545272] Lustre: server umount lustre-MDT0001 complete [ 6922.033768] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 6925.522814] Key type lgssc unregistered [ 6925.846503] LNet: 147320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6925.863045] LNetError: 147320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6925.891930] LNet: Removed LNI 192.168.202.146@tcp [ 6926.841212] Key type .llcrypt unregistered [ 6926.844862] Key type ._llcrypt unregistered [ 6948.130214] Key type ._llcrypt registered [ 6948.133450] Key type .llcrypt registered [ 6948.228475] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 6967.629259] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 6968.919298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6969.036082] alg: No test for adler32 (adler32-zlib) [ 6970.026262] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 6970.224499] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 6971.887209] Key type lgssc registered [ 6973.192709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7013.111830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7013.160750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7014.442728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7014.464660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7014.523707] Lustre: lustre-MDT0000: new disk, initializing [ 7014.594425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7014.621193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7019.718210] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7031.741029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.916799] Lustre: 150529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7031.974432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7031.978405] Lustre: Skipped 1 previous similar message [ 7032.036801] Lustre: lustre-MDT0001: new disk, initializing [ 7032.128405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7032.165936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7032.181293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7037.680931] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7047.977682] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7054.002684] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7061.425918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7061.706467] Lustre: lustre-OST0000: new disk, initializing [ 7061.717532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7061.729818] Lustre: 152496:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7061.823772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7069.065627] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7069.785921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7069.798600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7069.995348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7078.499419] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7085.278729] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_module obdecho/obdecho [ 7087.580982] LustreError: 153501:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7087.601672] LustreError: 153501:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7090.250963] LustreError: 153597:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7103.968376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7103.987356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7103.999258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7106.021164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7106.080326] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7106.106032] Lustre: Skipped 1 previous similar message [ 7109.971538] Lustre: server umount lustre-MDT0000 complete [ 7113.935348] LustreError: 150522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787461254 with bad export cookie 9274048487742758684 [ 7113.936258] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7113.955621] LustreError: 150522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7114.229512] Lustre: server umount lustre-MDT0001 complete [ 7130.354816] Lustre: server umount lustre-OST0000 complete [ 7156.735671] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 7173.897641] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7218.562268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7218.911179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7218.945727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7219.086649] Lustre: lustre-MDT0000: new disk, initializing [ 7219.208907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7219.236570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7224.042273] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7235.745416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7235.866878] Lustre: 157345:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7235.914581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7235.918588] Lustre: Skipped 1 previous similar message [ 7235.976625] Lustre: lustre-MDT0001: new disk, initializing [ 7236.045035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7236.082246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7236.094464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7241.517697] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7253.110486] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7259.789854] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7266.618639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7266.836724] Lustre: lustre-OST0000: new disk, initializing [ 7266.840742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7266.846802] Lustre: 159314:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7266.927336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7273.571013] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7273.577446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7273.730801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7274.094849] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7285.265152] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7290.419372] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7290.665481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7295.929405] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7296.214332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7299.559312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7299.565264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7299.576477] Lustre: Skipped 1 previous similar message [ 7299.584625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7299.618927] Lustre: Skipped 3 previous similar messages [ 7304.698720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7304.708048] Lustre: Skipped 1 previous similar message [ 7309.796935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7309.806195] Lustre: Skipped 1 previous similar message [ 7312.351748] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7312.793579] Lustre: server umount lustre-OST0000 complete [ 7317.990507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7318.001332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7318.022112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7323.661660] Lustre: server umount lustre-MDT0000 complete [ 7325.153532] LustreError: 157357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7325.161474] LustreError: 157357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7328.355592] LustreError: 157338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787461469 with bad export cookie 9274048487742759643 [ 7328.375607] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7328.379375] LustreError: 157338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7328.801229] Lustre: server umount lustre-MDT0001 complete [ 7333.565353] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:04:33 (1787461473) [ 7335.606288] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7338.813972] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 01:04:38 (1787461478) [ 7347.582647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7348.255091] LustreError: 161731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7348.368470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7353.835287] LustreError: 161732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7354.596495] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7358.946379] LustreError: 161731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7363.156863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7370.326761] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7381.486529] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7388.778337] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7397.301618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7397.701993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7397.710578] Lustre: Skipped 1 previous similar message [ 7405.621990] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7417.160938] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7428.726629] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 7428.742468] Lustre: Skipped 2 previous similar messages [ 7429.020049] Lustre: server umount lustre-MDT0000 complete [ 7429.611563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7429.622942] LustreError: 163190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7429.626648] Lustre: Skipped 2 previous similar messages [ 7429.658380] LustreError: 163190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7433.143619] LustreError: 161713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787461574 with bad export cookie 9274048487742760525 [ 7433.147691] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7433.161390] LustreError: 161713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7438.752629] LustreError: 161726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7438.773079] LustreError: 161726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7439.638184] Lustre: server umount lustre-MDT0001 complete [ 7447.760791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7462.714374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7463.101650] LustreError: 165575:0:(llog.c:1655:llog_backup()) MGC192.168.202.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7463.108117] Lustre: 165575:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7463.397651] LustreError: 165581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7463.418473] LustreError: 165581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7495.721244] LustreError: 165581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7495.745196] LustreError: 165581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 7523.807639] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7523.820733] Lustre: 165575:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7561.261115] LustreError: 165581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7561.276911] LustreError: 165581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 7564.800224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7568.894344] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7569.892782] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7578.610725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7593.444553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7604.708715] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x80b409f6ada7ffa6 [ 7604.717570] Lustre: MGC192.168.202.146@tcp: Connection restored to 0@lo (at 0@lo) [ 7610.349638] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7610.490770] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7619.829047] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7622.052090] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7628.571845] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7630.221504] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7651.296830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7651.299455] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7651.327906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7651.345384] Lustre: Skipped 4 previous similar messages [ 7651.362179] Lustre: Skipped 6 previous similar messages [ 7656.539055] Lustre: server umount lustre-MDT0000 complete [ 7661.399142] LustreError: 167317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787461802 with bad export cookie 9274048487742767014 [ 7661.419889] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7662.397559] Lustre: server umount lustre-MDT0001 complete [ 7667.698455] Lustre: server umount lustre-OST0000 complete [ 7694.993610] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 7711.677364] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7759.869185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7760.307012] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7760.368461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7760.506970] Lustre: lustre-MDT0000: new disk, initializing [ 7760.667371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7760.680677] Lustre: Skipped 1 previous similar message [ 7760.714370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7768.214473] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7784.381534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7784.565308] Lustre: 170717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7784.953505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7789.645946] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7800.798953] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7809.436415] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7817.771282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7818.011591] Lustre: lustre-OST0000: new disk, initializing [ 7818.014543] Lustre: Skipped 1 previous similar message [ 7818.019056] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7818.022964] Lustre: Skipped 2 previous similar messages [ 7818.028398] Lustre: 172688:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7824.800639] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7826.007166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7826.020940] Lustre: Skipped 1 previous similar message [ 7826.027849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7826.147671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7835.206151] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7841.267543] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7841.551508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7847.136796] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7847.425362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7852.013835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7852.014451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7852.033623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7852.053038] Lustre: Skipped 2 previous similar messages [ 7852.076589] Lustre: Skipped 2 previous similar messages [ 7864.288508] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7864.474805] Lustre: server umount lustre-OST0000 complete [ 7871.969991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7871.978102] LustreError: Skipped 1 previous similar message [ 7874.901606] Lustre: server umount lustre-MDT0000 complete [ 7877.614954] LustreError: 170722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7877.631974] LustreError: 170722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 7880.042876] LustreError: 170709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462021 with bad export cookie 9274048487742783814 [ 7880.050341] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7880.059636] LustreError: 170709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7880.613186] Lustre: server umount lustre-MDT0001 complete [ 7892.942812] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 01:13:52 (1787462032) [ 7901.448684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7907.385137] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7917.436118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7923.323812] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7932.502775] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7939.243611] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7947.617303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7954.784800] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7963.228617] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7976.294936] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 01:15:16 (1787462116) [ 7983.074039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7983.092663] Lustre: Skipped 2 previous similar messages [ 7983.109602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7983.112383] Lustre: Skipped 7 previous similar messages [ 7986.870991] Lustre: server umount lustre-OST0000 complete [ 7997.745756] Lustre: server umount lustre-MDT0000 complete [ 8001.762731] LustreError: 175084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462142 with bad export cookie 9274048487742784696 [ 8001.775066] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8002.338525] Lustre: server umount lustre-MDT0001 complete [ 8015.595329] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8018.559437] Key type lgssc unregistered [ 8018.850591] LNet: 178839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8018.854768] LNetError: 178839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8018.872691] LNet: Removed LNI 192.168.202.146@tcp [ 8019.868211] Key type .llcrypt unregistered [ 8019.870965] Key type ._llcrypt unregistered [ 8043.403489] Key type ._llcrypt registered [ 8043.409549] Key type .llcrypt registered [ 8043.645739] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8045.348203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8045.454299] alg: No test for adler32 (adler32-zlib) [ 8046.611695] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 8047.046687] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 8048.808244] Key type lgssc registered [ 8050.244488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8060.015756] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8060.031539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8075.551434] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8080.671936] LustreError: 180083:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.146@tcp: failed processing log, type 4: rc = -110 [ 8085.792604] LustreError: 180083:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8085.803279] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8085.811320] LustreError: MGC192.168.202.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8085.834207] LustreError: 180083:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8085.846517] LustreError: 180083:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8085.852932] LustreError: 180083:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8086.003972] Lustre: server umount lustre-OST0000 complete [ 8086.017375] LustreError: 180083:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8095.736641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8097.309421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8102.564772] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8107.499612] LustreError: 180582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8107.518790] LustreError: 180582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8112.509901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8112.609048] LustreError: 180583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8112.994786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8117.883404] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8128.358582] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8135.661949] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8144.178887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8144.346596] LustreError: 182398:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8144.362229] Lustre: 182398:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8144.521620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8151.445583] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8160.007894] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8162.170633] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8162.193489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8162.212427] Lustre: Skipped 1 previous similar message [ 8167.392708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8167.398740] Lustre: Skipped 4 previous similar messages [ 8168.619000] Lustre: server umount lustre-MDT0000 complete [ 8172.478073] LustreError: 180563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462313 with bad export cookie 9502146668532023646 [ 8172.494634] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8172.518326] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8172.520430] LustreError: 180578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8172.527831] Lustre: Skipped 1 previous similar message [ 8172.529185] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8172.792218] Lustre: server umount lustre-MDT0001 complete [ 8179.310586] Lustre: server umount lustre-OST0000 complete [ 8189.708606] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 8202.434867] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8248.108588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8248.398993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8248.432749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8248.579247] Lustre: lustre-MDT0000: new disk, initializing [ 8248.694369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8248.734184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8253.655380] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8264.790728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8264.954699] Lustre: 185924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8265.017336] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8265.021444] Lustre: Skipped 1 previous similar message [ 8265.142197] Lustre: lustre-MDT0001: new disk, initializing [ 8265.262286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8265.374090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8265.400263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8271.006698] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8282.890842] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8291.770222] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8300.262264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8300.692782] Lustre: lustre-OST0000: new disk, initializing [ 8300.702486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8300.715382] Lustre: 187895:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8300.814625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8306.754729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8306.767732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8306.876796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8308.065957] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8318.840303] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8324.400899] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8324.640395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8329.780677] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8330.073762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8332.770311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8332.775898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8332.793190] LustreError: Skipped 1 previous similar message [ 8332.794073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8332.837382] Lustre: Skipped 1 previous similar message [ 8337.907085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8337.918079] Lustre: Skipped 2 previous similar messages [ 8338.881340] Lustre: server umount lustre-OST0000 complete [ 8347.109459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8347.117088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8347.133339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8350.088633] Lustre: server umount lustre-MDT0000 complete [ 8353.248461] LustreError: 185935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8353.262343] LustreError: 185935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8354.662386] LustreError: 185915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462495 with bad export cookie 9502146668532024360 [ 8354.667918] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8354.672190] LustreError: 185915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8355.108715] Lustre: server umount lustre-MDT0001 complete [ 8369.729251] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 01:21:48 (1787462508) [ 8394.139477] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8398.576450] Key type lgssc unregistered [ 8398.964185] LNet: 190846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8398.980513] LNetError: 190846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8399.025560] LNet: Removed LNI 192.168.202.146@tcp [ 8400.425165] Key type .llcrypt unregistered [ 8400.430987] Key type ._llcrypt unregistered [ 8424.471471] Key type ._llcrypt registered [ 8424.473272] Key type .llcrypt registered [ 8424.558550] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8425.780535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8425.895188] alg: No test for adler32 (adler32-zlib) [ 8426.989392] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 8427.329710] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 8429.168870] Key type lgssc registered [ 8430.554866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8438.881788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8438.903989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8440.313770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8444.937654] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8450.535563] LustreError: 192064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8450.579603] LustreError: 192064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8454.415231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8454.956070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8461.752104] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8474.100910] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8482.612313] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8491.319446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8491.487135] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8491.493604] LustreError: 193929:0:(llog.c:1696:llog_backup()) MGC192.168.202.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8491.497234] Lustre: 193929:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8491.596280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8499.196185] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8508.182488] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8511.459496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8511.480236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8511.527165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8515.057798] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8515.062319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8515.087753] Lustre: Skipped 1 previous similar message [ 8515.103354] Lustre: Skipped 2 previous similar messages [ 8517.066067] Lustre: server umount lustre-MDT0000 complete [ 8520.169407] LustreError: 192060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8520.203224] LustreError: 192060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8522.244392] LustreError: 192043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462663 with bad export cookie 1310079224134115989 [ 8522.272840] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8522.615873] Lustre: server umount lustre-MDT0001 complete [ 8537.619061] Lustre: server umount lustre-OST0000 complete [ 8547.820807] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 8560.024597] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8602.887671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8603.087796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8603.123205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8603.208721] Lustre: lustre-MDT0000: new disk, initializing [ 8603.300413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8603.328692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8607.342770] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8618.678490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8618.806193] Lustre: 197456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8618.851906] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8618.853900] Lustre: Skipped 1 previous similar message [ 8618.856616] LustreError: 198240:0:(llog.c:1696:llog_backup()) MGC192.168.202.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8618.862017] LustreError: 198240:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8618.865803] Lustre: 198240:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8618.879820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8618.883329] Lustre: Skipped 1 previous similar message [ 8618.954352] Lustre: lustre-MDT0001: new disk, initializing [ 8619.134462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8619.181925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8619.209369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8625.211847] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8637.238354] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8646.236783] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8654.876221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8655.121430] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8655.123011] Lustre: Skipped 1 previous similar message [ 8655.125405] LustreError: 199402:0:(llog.c:1696:llog_backup()) MGC192.168.202.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8655.130673] LustreError: 199402:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8655.134479] Lustre: 199402:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8655.139655] Lustre: 199402:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8655.159639] Lustre: lustre-OST0000: new disk, initializing [ 8655.162444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8655.166824] Lustre: 199426:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8655.233954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8661.302331] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8665.212985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8665.228327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8665.354660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8671.996659] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8677.797866] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8678.105089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8683.573729] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8683.910839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8691.176114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8691.189286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8691.216339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8691.219751] Lustre: Skipped 1 previous similar message [ 8696.289703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8696.298132] Lustre: Skipped 1 previous similar message [ 8701.410713] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8701.420406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8701.435612] Lustre: Skipped 1 previous similar message [ 8701.743540] Lustre: server umount lustre-OST0000 complete [ 8706.529705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8706.542762] Lustre: Skipped 1 previous similar message [ 8711.136843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8711.143029] LustreError: Skipped 1 previous similar message [ 8711.147687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8711.168447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8711.180302] Lustre: Skipped 1 previous similar message [ 8712.675631] Lustre: server umount lustre-MDT0000 complete [ 8716.774508] LustreError: 197470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8716.807732] LustreError: 197470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8718.249045] LustreError: 197449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462859 with bad export cookie 1310079224134116710 [ 8718.264404] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8718.266397] LustreError: 197449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8719.059958] Lustre: server umount lustre-MDT0001 complete [ 8730.491919] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 01:27:50 (1787462870) [ 8755.813209] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8767.988823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8768.330636] LustreError: 202578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8768.393394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8773.611522] LustreError: 202579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8775.591966] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8783.860606] LustreError: 202579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8783.881780] LustreError: 202579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8785.502798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8791.575921] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8794.215141] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8802.923232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8803.153517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8803.162707] Lustre: Skipped 1 previous similar message [ 8811.715375] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8823.712224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8823.958327] Lustre: lustre-OST0001: new disk, initializing [ 8823.962362] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8823.967182] Lustre: 204888:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8830.567486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8830.578875] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8830.685629] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8831.657868] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8843.761350] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8855.321674] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8857.746920] Lustre: DEBUG MARKER: ADD OST3 [ 8863.880235] loop: module loaded [ 8863.986658] loop0: detected capacity change from 0 to 204800000 [ 8864.318366] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8864.340833] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8864.363923] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8872.009776] loop0: detected capacity change from 0 to 204800000 [ 8872.014677] Lustre: DEBUG MARKER: STOP OST3 [ 8879.296142] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8879.650175] Lustre: lustre-OST0003: new disk, initializing [ 8879.654993] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8879.676140] Lustre: 206942:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8879.863392] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8879.883592] Lustre: Skipped 1 previous similar message [ 8881.165302] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8881.436177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8881.450197] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8881.513735] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8883.232563] LustreError: 207032:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8883.257234] Lustre: Failing over lustre-OST0003 [ 8883.456316] Lustre: server umount lustre-OST0003 complete [ 8886.756112] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8886.769604] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8886.796441] LustreError: 204065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8887.897608] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8898.928315] Lustre: DEBUG MARKER: START OST3 again [ 8905.998127] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8906.405918] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8907.307867] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8907.617132] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8907.617299] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8912.303910] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8934.613639] Lustre: Failing over lustre-OST0003 [ 8934.722512] Lustre: server umount lustre-OST0003 complete [ 8937.465688] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8937.487302] Lustre: Skipped 2 previous similar messages [ 8937.491637] LustreError: 204066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8937.552643] LustreError: 204066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8957.923116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8957.936141] Lustre: Skipped 2 previous similar messages [ 8962.753503] Lustre: server umount lustre-MDT0000 complete [ 8967.251048] LustreError: 202559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463108 with bad export cookie 1310079224134117592 [ 8967.254357] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8967.258836] LustreError: 202559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8974.181819] Lustre: server umount lustre-MDT0001 complete [ 8979.476201] Lustre: server umount lustre-OST0000 complete [ 8983.463429] Lustre: server umount lustre-OST0001 complete [ 8991.493339] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 9003.282247] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9048.259134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9048.746219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9048.796413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9048.927376] Lustre: lustre-MDT0000: new disk, initializing [ 9049.040991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9049.045755] Lustre: Skipped 1 previous similar message [ 9049.080759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9055.639605] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9070.246034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9070.373630] Lustre: 212505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9070.432118] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9070.434771] Lustre: Skipped 1 previous similar message [ 9070.489719] Lustre: lustre-MDT0001: new disk, initializing [ 9070.588471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9070.601723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9077.337792] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9090.253570] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9099.737687] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9109.669662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9110.244749] Lustre: lustre-OST0000: new disk, initializing [ 9110.249893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9110.258752] Lustre: 214476:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9111.795369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9111.822860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9112.030488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9120.884719] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9135.922320] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9143.069595] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9143.425448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9148.445765] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9148.694552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9152.996101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9153.001178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9153.015279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9153.015287] Lustre: Skipped 4 previous similar messages [ 9153.038507] Lustre: Skipped 7 previous similar messages [ 9165.279166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9165.670579] Lustre: server umount lustre-OST0000 complete [ 9172.961055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9172.990875] LustreError: Skipped 1 previous similar message [ 9185.247417] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9185.531828] Lustre: server umount lustre-MDT0000 complete [ 9188.837396] LustreError: 214484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9188.849502] LustreError: 214484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 9190.550793] LustreError: 212498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463331 with bad export cookie 1310079224134119888 [ 9190.575400] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9190.577941] LustreError: 212498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9191.518414] Lustre: server umount lustre-MDT0001 complete [ 9197.072401] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 01:35:36 (1787463336) [ 9216.525981] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 9233.407554] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9281.205990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9281.469608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9281.506478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9281.580658] Lustre: lustre-MDT0000: new disk, initializing [ 9281.733815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9287.253833] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9299.797163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.954107] Lustre: 219117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9300.307807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9305.212154] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9316.702658] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9323.055508] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9331.205990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9331.588456] Lustre: 221087:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9331.716547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9331.721804] Lustre: Skipped 4 previous similar messages [ 9333.644716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9333.859770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9339.169282] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9350.584436] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9356.530817] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9356.837189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9363.940593] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9364.278388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9369.570392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9369.571021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9369.598526] Lustre: Skipped 3 previous similar messages [ 9369.623403] Lustre: Skipped 13 previous similar messages [ 9372.922352] Lustre: server umount lustre-OST0000 complete [ 9377.249494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9383.018617] Lustre: server umount lustre-MDT0000 complete [ 9384.930545] LustreError: 219996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9384.937960] LustreError: 219996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9386.914350] LustreError: 219111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463527 with bad export cookie 1310079224134120770 [ 9386.923316] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9386.927189] LustreError: 219111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9387.263318] Lustre: server umount lustre-MDT0001 complete [ 9396.065889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9401.546331] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9411.197531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9417.098967] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9425.612770] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9431.376818] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9439.826719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9446.675188] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9455.059954] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9466.869778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9472.404917] Lustre: server umount lustre-OST0000 complete [ 9483.150477] Lustre: server umount lustre-MDT0000 complete [ 9487.623808] LustreError: 223288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463628 with bad export cookie 1310079224134121652 [ 9487.633337] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9487.635484] LustreError: 223288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9488.010087] Lustre: server umount lustre-MDT0001 complete [ 9595.738475] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 9598.966108] Key type lgssc unregistered [ 9599.246570] LNet: 226971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9599.251332] LNetError: 226971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9600.298580] LNet: Removed LNI 192.168.202.146@tcp [ 9601.359380] Key type .llcrypt unregistered [ 9601.362507] Key type ._llcrypt unregistered [ 9632.578949] Key type ._llcrypt registered [ 9632.584169] Key type .llcrypt registered [ 9632.716520] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 9655.417958] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9658.324183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9658.352078] alg: No test for adler32 (adler32-zlib) [ 9660.020454] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 9660.641307] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 9662.567157] Key type lgssc registered [ 9665.332641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9708.920392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9708.967140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9710.491364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9710.569140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9710.740206] Lustre: lustre-MDT0000: new disk, initializing [ 9710.816156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9710.837749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9715.871439] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9728.237538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9728.387820] Lustre: 230330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9728.469449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9728.476442] Lustre: Skipped 1 previous similar message [ 9728.600835] Lustre: lustre-MDT0001: new disk, initializing [ 9728.703506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9728.743316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9728.759471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9733.676911] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9746.873774] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9752.508200] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9759.392676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9759.642174] Lustre: lustre-OST0000: new disk, initializing [ 9759.647266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9759.653609] Lustre: 232297:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9759.737501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9764.940595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9764.959240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9765.181839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9765.829363] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9775.163780] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9779.978084] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9780.125950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9784.053954] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9784.271805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9790.961536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9790.975802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9790.978418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9790.988610] Lustre: Skipped 1 previous similar message [ 9791.037558] LustreError: Skipped 1 previous similar message [ 9792.701281] Lustre: server umount lustre-OST0000 complete [ 9796.065514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9796.066305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9796.072380] Lustre: Skipped 1 previous similar message [ 9796.080684] Lustre: Skipped 1 previous similar message [ 9801.191176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9801.208699] Lustre: Skipped 2 previous similar messages [ 9802.933704] Lustre: server umount lustre-MDT0000 complete [ 9807.475320] LustreError: 230323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463948 with bad export cookie 13355585312664748954 [ 9807.478028] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9807.484509] LustreError: 230323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9807.895943] Lustre: server umount lustre-MDT0001 complete [ 9829.340415] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9841.351282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9841.977807] LustreError: 235232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9842.002773] LustreError: 235232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9842.040426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9847.285765] LustreError: 235233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9847.314459] LustreError: 235233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9848.078874] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9852.385783] LustreError: 235232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9857.514753] LustreError: 235233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9860.849088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9861.460788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9867.390939] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9870.579280] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9878.830098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9879.226819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9887.417836] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9898.326557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9898.627754] Lustre: lustre-OST0001: new disk, initializing [ 9898.632267] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9898.644706] Lustre: 237538:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9898.782250] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9906.369650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9906.390465] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9906.533750] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9906.575639] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9920.661623] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9928.885816] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9943.661301] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 01:48:03 (1787464083) [ 9947.625677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9947.630636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9947.647184] Lustre: Skipped 3 previous similar messages [ 9947.667084] Lustre: Skipped 3 previous similar messages [ 9952.242533] Lustre: server umount lustre-MDT0000 complete [ 9952.739052] LustreError: 235227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9956.320234] LustreError: 235213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787464097 with bad export cookie 13355585312664749836 [ 9956.332866] LustreError: 235213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9956.339287] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9956.699780] Lustre: server umount lustre-MDT0001 complete [ 9971.513807] Lustre: server umount lustre-OST0000 complete [ 9973.728266] Lustre: 228553:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787464098/real 1787464098] req@ffff98114530d880 x1874291642714752/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787464114 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9973.765899] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9975.662734] Lustre: server umount lustre-OST0001 complete [ 9982.621992] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 9995.955490] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [10048.083040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10048.418804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10048.469445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10048.568510] Lustre: lustre-MDT0000: new disk, initializing [10048.661612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10048.674859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10053.000419] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10064.550498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10064.631728] Lustre: 241490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10064.673979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10064.678273] Lustre: Skipped 1 previous similar message [10064.732262] Lustre: lustre-MDT0001: new disk, initializing [10064.832231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10064.842698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10069.486938] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10080.710886] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10087.703348] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10094.894586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10095.102257] Lustre: lustre-OST0000: new disk, initializing [10095.105617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10095.115495] Lustre: 243455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10096.555890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10096.571759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10096.788716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10103.989937] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10116.071699] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10120.614311] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10120.904829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10125.851613] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10126.023445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10132.465664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10132.466778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10132.482532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10132.487367] Lustre: Skipped 1 previous similar message [10142.175109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10142.336326] Lustre: server umount lustre-OST0000 complete [10146.784742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10146.795739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10146.806625] Lustre: Skipped 1 previous similar message [10146.811532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10146.814249] Lustre: Skipped 2 previous similar messages [10152.614844] Lustre: server umount lustre-MDT0000 complete [10152.929171] LustreError: 241501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10152.953643] LustreError: 241501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10158.719513] LustreError: 241481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787464299 with bad export cookie 13355585312664751971 [10158.740161] LustreError: 241481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10158.745592] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10159.314767] Lustre: server umount lustre-MDT0001 complete [10171.348209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10171.808495] LustreError: 245674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10171.825021] LustreError: 245674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10171.887289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10171.892977] Lustre: Skipped 2 previous similar messages [10177.098839] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10187.565593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10193.679939] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10203.978700] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10210.891595] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10219.737488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10227.749714] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10235.991290] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10385.376258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10385.387908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10385.391328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10385.391336] Lustre: Skipped 2 previous similar messages [10385.428627] Lustre: Skipped 2 previous similar messages [10391.279833] Lustre: server umount lustre-OST0000 complete [10398.176583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10402.836690] Lustre: server umount lustre-MDT0000 complete [10405.857630] LustreError: 245674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10405.885824] LustreError: 245674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10407.664280] LustreError: 245655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787464548 with bad export cookie 13355585312664752853 [10407.678510] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10407.694183] LustreError: 245655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10408.277325] Lustre: server umount lustre-MDT0001 complete [10507.125933] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [10510.795223] Key type lgssc unregistered [10511.137683] LNet: 249290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10511.148253] LNetError: 249290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10512.165702] LNet: Removed LNI 192.168.202.146@tcp [10513.523573] Key type .llcrypt unregistered [10513.527531] Key type ._llcrypt unregistered [10547.722565] Key type ._llcrypt registered [10547.724212] Key type .llcrypt registered [10547.865059] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [10569.732064] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [10571.690233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10571.779019] alg: No test for adler32 (adler32-zlib) [10572.996156] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [10573.352090] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [10575.095170] Key type lgssc registered [10576.500812] Lustre: Echo OBD driver; http://www.lustre.org/ [10638.781862] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [10653.014883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10653.063585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10654.475273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10654.509094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10654.594949] Lustre: lustre-MDT0000: new disk, initializing [10654.764884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10654.808516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10660.857805] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10674.800103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10674.944340] Lustre: 253389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10674.999411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10675.003596] Lustre: Skipped 1 previous similar message [10675.132381] Lustre: lustre-MDT0001: new disk, initializing [10675.257798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10675.323863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10675.339577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10681.211332] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10686.669828] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10695.271844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10695.623267] Lustre: lustre-OST0000: new disk, initializing [10695.627107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10695.636214] Lustre: 255020:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10695.733727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10696.156605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10696.195546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10696.321326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10706.022966] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10722.969370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10723.369048] Lustre: lustre-OST0001: new disk, initializing [10723.371852] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10723.391778] Lustre: 255907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10723.626875] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10730.155091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10730.168303] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10730.288435] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10733.371755] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10747.798887] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10756.298706] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10772.451353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10772.478201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10772.521685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10776.551264] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10776.555619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10776.582670] Lustre: Skipped 2 previous similar messages [10776.593086] Lustre: Skipped 3 previous similar messages [10778.548613] Lustre: server umount lustre-MDT0000 complete [10783.015042] LustreError: 254184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787464924 with bad export cookie 7254689460491799275 [10783.017511] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10783.027824] LustreError: 254184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10783.402651] Lustre: server umount lustre-MDT0001 complete [10798.889506] Lustre: server umount lustre-OST0000 complete [10802.670133] Lustre: 250874:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787464927/real 1787464927] req@ffff98127d725180 x1874292600036224/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787464943 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10802.712492] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10804.191240] Lustre: 250872:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787464929/real 1787464929] req@ffff98114eaace00 x1874292600036480/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787464945 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10805.243463] Lustre: server umount lustre-OST0001 complete [10823.417751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10836.858927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10847.967185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10860.236499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10885.399585] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [10898.363555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10898.396377] Lustre: lustre-MDT0000: reset Object Index mappings [10898.924763] LustreError: 260920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10898.936914] LustreError: 260920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10898.989157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10904.032237] LustreError: 260921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10904.207358] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10909.161242] LustreError: 260920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10914.275121] LustreError: 260921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10914.334532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10914.376113] Lustre: lustre-MDT0001: reset Object Index mappings [10914.995851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10920.142726] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10924.318370] Lustre: 262063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10933.516427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10933.909671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10939.075716] LustreError: 262416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10942.085325] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10944.257631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10949.608769] LustreError: 262419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10949.632413] LustreError: 262419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10952.258633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10957.812802] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10960.484478] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10969.576660] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10974.849828] Lustre: 263937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10983.396828] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10983.419816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10983.436604] Lustre: Skipped 1 previous similar message [10983.471931] Lustre: Skipped 3 previous similar messages [10988.519872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10988.535972] Lustre: Skipped 3 previous similar messages [10989.251433] Lustre: server umount lustre-MDT0000 complete [10993.566253] LustreError: 260900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787465134 with bad export cookie 7254689460491806065 [10993.572824] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10993.587043] LustreError: 260900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10993.633118] LustreError: 262438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10993.634808] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10993.648450] LustreError: 262438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10993.658728] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10993.663713] Lustre: Skipped 3 previous similar messages [10993.685302] Lustre: Skipped 1 previous similar message [10994.115325] Lustre: server umount lustre-MDT0001 complete [10999.330252] Lustre: server umount lustre-OST0000 complete [11004.351347] Lustre: server umount lustre-OST0001 complete [11029.653442] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 02:06:09 (1787465169) [11061.831263] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [11079.112055] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [11128.003259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11128.406561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11128.465207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11128.546442] Lustre: lustre-MDT0000: new disk, initializing [11128.618700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11128.623696] Lustre: Skipped 1 previous similar message [11128.642954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11134.698514] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11147.310684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11147.491348] Lustre: 268220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11147.567336] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11147.577401] Lustre: Skipped 1 previous similar message [11147.690914] Lustre: lustre-MDT0001: new disk, initializing [11147.828279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11147.840349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11153.301832] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11166.074962] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11173.499197] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11183.126670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11183.853198] Lustre: lustre-OST0000: new disk, initializing [11183.876468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11183.903097] Lustre: 270189:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11185.867932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11185.880181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11186.118739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11195.978724] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11208.585236] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11213.821549] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11214.059299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11218.742189] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11219.007340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11221.984267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11221.989034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11221.997744] LustreError: Skipped 1 previous similar message [11222.017681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11222.023697] Lustre: Skipped 1 previous similar message [11222.038776] Lustre: Skipped 1 previous similar message [11235.295169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11235.509181] Lustre: server umount lustre-OST0000 complete [11245.025308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11245.048655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11245.082279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11245.085162] Lustre: Skipped 6 previous similar messages [11255.263315] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11255.701018] Lustre: server umount lustre-MDT0000 complete [11255.788662] LustreError: 268231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11260.833918] LustreError: 268212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787465401 with bad export cookie 7254689460491808158 [11260.838842] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11260.858926] LustreError: 268212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11261.442749] Lustre: server umount lustre-MDT0001 complete [11285.616421] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [11299.693671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11300.004429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11300.011521] Lustre: Skipped 2 previous similar messages [11303.632132] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11313.267514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11318.933642] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11321.974301] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11329.364612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11337.370203] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11347.922162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11348.243270] Lustre: lustre-OST0001: new disk, initializing [11348.269175] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11348.283314] Lustre: 275438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11350.056679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11350.062904] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11350.169274] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11355.998179] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11367.378891] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11371.859837] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11378.347600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11378.353472] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11390.433910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11390.444860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11390.468234] Lustre: Skipped 1 previous similar message [11390.474351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11390.477138] Lustre: Skipped 3 previous similar messages [11393.855847] Lustre: server umount lustre-MDT0000 complete [11396.593654] LustreError: 274090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11396.620616] LustreError: 274090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11399.163345] LustreError: 273111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787465540 with bad export cookie 7254689460491809040 [11399.180810] LustreError: 273111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11399.181608] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11399.865992] Lustre: server umount lustre-MDT0001 complete [11415.632876] Lustre: server umount lustre-OST0000 complete [11417.887875] Lustre: 250874:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787465542/real 1787465542] req@ffff9811436c9880 x1874292600208640/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787465558 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11420.142440] Lustre: 250872:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787465545/real 1787465545] req@ffff98114f327b80 x1874292600208896/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787465561 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11420.733492] Lustre: server umount lustre-OST0001 complete [11444.794921] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [11457.342530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11457.953542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11457.958468] Lustre: Skipped 3 previous similar messages [11465.061947] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11478.015990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11484.208259] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11488.092168] Lustre: 279639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11497.692795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11502.476918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11506.782235] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11520.030555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11525.639536] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11529.703978] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11539.379990] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11545.382811] Lustre: 281516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11552.188416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11552.267085] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11565.452425] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 02:15:05 (1787465705) [11586.017272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11586.043317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11586.067249] Lustre: Skipped 4 previous similar messages [11586.076949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11586.088494] Lustre: Skipped 4 previous similar messages [11588.514410] Lustre: server umount lustre-MDT0000 complete [11592.165486] LustreError: 278494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11592.184840] LustreError: 278494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [11592.283842] LustreError: 278479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787465733 with bad export cookie 7254689460491815599 [11592.292754] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11592.300107] LustreError: 278479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11598.810285] Lustre: server umount lustre-MDT0001 complete [11609.660819] Lustre: server umount lustre-OST0000 complete [11614.317706] Lustre: server umount lustre-OST0001 complete [11622.011799] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [11636.936512] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [11695.668311] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [11709.061839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11709.395615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11709.458165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11709.587983] Lustre: lustre-MDT0000: new disk, initializing [11709.711527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11715.366467] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11729.353540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11729.483337] Lustre: 285780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11729.816686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11729.827548] Lustre: Skipped 4 previous similar messages [11729.886356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11736.956842] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11742.786684] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11751.331663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11751.679410] Lustre: 287415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11754.084402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11759.244542] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11772.246307] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11772.473454] Lustre: 288284:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11773.735646] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11773.754526] Lustre: Skipped 1 previous similar message [11773.910321] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11781.110539] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11794.101427] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11821.352832] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 02:19:20 (1787465960) [11825.129572] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11825.137970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11825.169939] Lustre: Skipped 6 previous similar messages [11825.183415] Lustre: Skipped 6 previous similar messages [11830.788757] Lustre: server umount lustre-MDT0000 complete [11836.713088] LustreError: 285774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787465977 with bad export cookie 7254689460491823243 [11836.728565] LustreError: 285774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11836.731971] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11837.528892] Lustre: server umount lustre-MDT0001 complete [11854.986814] Lustre: server umount lustre-OST0000 complete [11856.927159] Lustre: 250873:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787465981/real 1787465981] req@ffff98125e93c000 x1874292600380032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787465997 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11860.167705] Lustre: server umount lustre-OST0001 complete [11871.442255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11871.932256] LustreError: 290535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11871.951493] LustreError: 290535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11877.524321] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11889.613154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11895.658859] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11905.457601] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11912.901469] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11921.023077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11928.414922] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11937.777536] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11965.558158] loop1: detected capacity change from 0 to 204800000 [11965.877573] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11965.895307] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11965.932869] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11972.661491] loop1: detected capacity change from 0 to 204800000 [11982.340911] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11982.599733] LustreError: 290525:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs32002: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11982.627992] LustreError: 290525:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs32002 [11982.644480] LustreError: 290525:0:(mgs_handler.c:517:mgs_target_reg()) fs32002-MDT0001: Error checking target index: rc = -13 [11982.684047] LustreError: fs32002-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11982.700186] LustreError: 294258:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11982.708466] LustreError: 294258:0:(tgt_mount.c:2129:server_put_super()) no obd fs32002-MDT0001 [11982.723126] LustreError: 294258:0:(tgt_mount.c:116:server_deregister_mount()) fs32002-MDT0001 not registered [11982.789235] Lustre: server umount fs32002-MDT0001 complete [11982.793083] LustreError: 294258:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11995.791220] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11995.923943] Lustre: Setting parameter fs32002-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs32002-MDT0001 [11995.937599] Lustre: Skipped 1 previous similar message [11996.017788] Lustre: srv-fs32002-MDT0001: No data found on store. Initialize space. [11996.029520] Lustre: Skipped 4 previous similar messages [11996.169389] Lustre: fs32002-MDT0001: new disk, initializing [11996.180505] Lustre: Skipped 3 previous similar messages [11996.466219] Lustre: srv-fs32002-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11996.478652] Lustre: 294725:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32002-MDT0001-mdtlov: can't create new llog: rc = -115 [11996.501690] LustreError: 294725:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32002-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12002.702321] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12012.546931] Lustre: server umount fs32002-MDT0001 complete [12014.574205] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12033.585886] loop1: detected capacity change from 0 to 204800000 [12042.509533] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12043.282701] Lustre: 296297:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32002-MDT0001-mdtlov: can't create new llog: rc = -115 [12043.300535] LustreError: 296297:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32002-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12049.741657] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12055.092268] Lustre: server umount fs32002-MDT0001 complete [12069.345092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12069.361060] LustreError: Skipped 1 previous similar message [12073.187693] Lustre: server umount lustre-MDT0000 complete [12084.347544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12084.479570] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12089.908055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12092.083212] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12102.811693] loop1: detected capacity change from 0 to 204800000 [12109.521924] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12110.017723] Lustre: 298747:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32002-MDT0001-mdtlov: can't create new llog: rc = -115 [12110.036271] LustreError: 298747:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32002-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12116.761761] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12123.771223] Lustre: server umount fs32002-MDT0001 complete [12154.860060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12154.879105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12154.890171] Lustre: Skipped 6 previous similar messages [12154.895532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12154.902794] Lustre: Skipped 13 previous similar messages [12169.183176] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12169.564353] Lustre: server umount lustre-OST0000 complete [12189.153475] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12189.531708] Lustre: server umount lustre-MDT0000 complete [12194.675292] LustreError: 290516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787466335 with bad export cookie 7254689460491825924 [12194.693160] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12194.697219] LustreError: 290516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12195.005404] Lustre: server umount lustre-MDT0001 complete [12210.641956] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [12214.676327] Key type lgssc unregistered [12215.088691] LNet: 301100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12215.098868] LNetError: 301100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12215.125868] LNet: Removed LNI 192.168.202.146@tcp [12216.469172] Key type .llcrypt unregistered [12216.470758] Key type ._llcrypt unregistered [12240.573263] Key type ._llcrypt registered [12240.577855] Key type .llcrypt registered [12240.701376] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [12242.899938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12243.172450] alg: No test for adler32 (adler32-zlib) [12244.557984] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [12244.968474] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [12246.695749] Key type lgssc registered [12248.574885] Lustre: Echo OBD driver; http://www.lustre.org/ [12260.021258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12260.060757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12261.788110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12268.849458] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12271.077107] LustreError: 302373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12271.120920] LustreError: 302373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12276.210607] LustreError: 302374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12280.118527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12280.769480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12285.842288] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12297.339496] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12306.119924] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12316.252477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12316.622871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12324.358608] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12325.807113] LustreError: 304207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12330.976899] LustreError: 304210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12334.814746] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12336.985905] LustreError: 304210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12337.020212] LustreError: 304210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12347.385335] LustreError: 304207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12347.403684] LustreError: 304207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [12356.950423] loop1: detected capacity change from 0 to 204800000 [12357.180310] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12357.192672] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12357.222386] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12364.253994] loop1: detected capacity change from 0 to 204800000 [12367.860317] LustreError: 304636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12367.887873] LustreError: 304636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [12371.316610] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12371.512886] Lustre: fs9159-OST0000: new disk, initializing [12371.516697] Lustre: srv-fs9159-OST0000: No data found on store. Initialize space. [12371.524088] Lustre: 306112:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12371.627243] Lustre: fs9159-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12380.454621] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12391.356129] Lustre: server umount fs9159-OST0000 complete [12403.694527] LustreError: 304209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12403.703508] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12403.706637] LustreError: 304209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [12407.168394] loop1: detected capacity change from 0 to 204800000 [12408.814404] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12414.218287] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12414.419348] Lustre: fs9159-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12421.870374] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12433.927430] loop2: detected capacity change from 0 to 204800000 [12434.273164] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12434.305103] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12434.339799] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12440.756110] loop2: detected capacity change from 0 to 204800000 [12447.927103] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12448.103099] LustreError: fs9159-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12448.115886] LustreError: fs9159-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12448.124245] LustreError: 309158:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12448.130306] LustreError: 309158:0:(tgt_mount.c:2129:server_put_super()) no obd fs9159-OST0001 [12448.134768] LustreError: 309158:0:(tgt_mount.c:116:server_deregister_mount()) fs9159-OST0001 not registered [12448.172382] Lustre: server umount fs9159-OST0001 complete [12448.183777] LustreError: 309158:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12460.867097] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12460.990360] Lustre: fs9159-OST0001: new disk, initializing [12460.993363] Lustre: srv-fs9159-OST0001: No data found on store. Initialize space. [12460.998123] Lustre: 309710:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12461.085269] Lustre: fs9159-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12470.013473] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12470.255705] LustreError: 304209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12470.289261] LustreError: 304209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [12481.041528] Lustre: server umount fs9159-OST0000 complete [12493.067341] Lustre: server umount fs9159-OST0001 complete [12508.271705] loop1: detected capacity change from 0 to 204800000 [12515.546630] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12515.764967] Lustre: fs9159-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12524.740775] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12533.682942] loop2: detected capacity change from 0 to 204800000 [12540.659807] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12550.770040] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12557.306036] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12562.450485] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12564.156924] Lustre: server umount fs9159-OST0000 complete [12576.189174] Lustre: server umount fs9159-OST0001 complete [12597.220211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12597.231083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12597.259884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12598.243872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12598.245648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12599.513429] Lustre: server umount lustre-OST0000 complete [12608.482769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12608.493398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12608.504690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12608.528138] Lustre: Skipped 2 previous similar messages [12611.434033] Lustre: server umount lustre-MDT0000 complete [12613.604650] LustreError: 306941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12613.631107] LustreError: 306941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 75 previous similar messages [12616.682533] LustreError: 302354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787466757 with bad export cookie 6137893309270796458 [12616.683816] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12616.687744] LustreError: 302354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12617.223041] Lustre: server umount lustre-MDT0001 complete [12632.179185] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [12636.481529] Key type lgssc unregistered [12636.803609] LNet: 315640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12636.812485] LNetError: 315640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12636.837981] LNet: Removed LNI 192.168.202.146@tcp [12637.662127] Key type .llcrypt unregistered [12637.664242] Key type ._llcrypt unregistered [12677.406632] Key type ._llcrypt registered [12677.408292] Key type .llcrypt registered [12677.582122] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [12701.255703] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [12703.468883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12703.691233] alg: No test for adler32 (adler32-zlib) [12704.978865] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [12705.406522] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [12707.127177] Key type lgssc registered [12708.559728] Lustre: Echo OBD driver; http://www.lustre.org/ [12753.205810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12753.223363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12754.513558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12754.576479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12754.744021] Lustre: lustre-MDT0000: new disk, initializing [12754.847201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12754.883321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12760.553600] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12774.283328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12774.459956] Lustre: 319506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12774.562877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12774.566755] Lustre: Skipped 1 previous similar message [12774.634253] Lustre: lustre-MDT0001: new disk, initializing [12774.744256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12774.795997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12774.803742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12779.841845] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12791.185131] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12799.058878] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12808.411783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12808.979205] Lustre: lustre-OST0000: new disk, initializing [12809.008706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12809.027588] Lustre: 321473:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12809.178672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12816.530551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12816.553826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12816.750842] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12817.774667] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12830.900583] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12833.994479] LustreError: 317677:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [12834.081031] Lustre: Mounted lustre-client - version 2.17.57_63_gaeb0ce0 [12835.367262] Lustre: Unmounted lustre-client [12847.591051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12847.606239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12847.627425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12849.636068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12849.649891] Lustre: Skipped 2 previous similar messages [12853.830445] Lustre: server umount lustre-OST0000 complete [12859.872629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12859.875866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12859.900069] Lustre: Skipped 2 previous similar messages [12859.919511] Lustre: Skipped 1 previous similar message [12864.993515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12865.012755] Lustre: Skipped 1 previous similar message [12865.742333] Lustre: server umount lustre-MDT0000 complete [12870.289519] LustreError: 321472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787467011 with bad export cookie 1654217035285170749 [12870.297052] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12870.304959] LustreError: 321472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12870.629157] Lustre: server umount lustre-MDT0001 complete [12887.318099] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [12887.712086] LNet: 1 local NIs in recovery (showing 1): 192.168.202.146@tcp [12892.734854] Key type lgssc unregistered [12893.118837] LNet: 323638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12893.134561] LNetError: 323638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12965.871881] LNet: Removed LNI 192.168.202.146@tcp [12967.051331] Key type .llcrypt unregistered [12967.059120] Key type ._llcrypt unregistered [12992.170675] Key type ._llcrypt registered [12992.172861] Key type .llcrypt registered [12992.459893] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [13014.593359] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [13015.918296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13015.950606] alg: No test for adler32 (adler32-zlib) [13017.353173] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [13017.743814] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [13019.503251] Key type lgssc registered [13020.782835] Lustre: Echo OBD driver; http://www.lustre.org/ [13061.106958] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:40:00 (1787467200) [13069.606517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13069.672154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13071.008317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13071.051667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13071.154828] Lustre: lustre-MDT0000: new disk, initializing [13071.284783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13071.326678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13076.805517] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13089.578557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13089.791755] Lustre: 326862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13089.873829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13089.878100] Lustre: Skipped 1 previous similar message [13089.931884] Lustre: lustre-MDT0001: new disk, initializing [13090.044154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13090.081468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13090.097430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13094.728204] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13109.452471] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13118.924219] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13128.087663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13128.614629] Lustre: lustre-OST0000: new disk, initializing [13128.623161] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13128.632486] Lustre: 328831:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13128.791357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13135.413524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13135.432341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13135.523142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13137.364809] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13149.987350] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13171.684447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13171.686977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13171.709555] Lustre: Skipped 1 previous similar message [13171.721523] Lustre: Skipped 1 previous similar message [13176.574502] Lustre: server umount lustre-OST0000 complete [13181.932080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13181.950586] Lustre: Skipped 1 previous similar message [13181.957440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13187.340287] Lustre: server umount lustre-MDT0000 complete [13192.096588] LustreError: 326855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787467333 with bad export cookie 10847071841370347554 [13192.109429] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13192.117360] LustreError: 326855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13192.574808] Lustre: server umount lustre-MDT0001 complete [13205.986801] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13209.375630] Key type lgssc unregistered [13209.776721] LNet: 330745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13209.784485] LNetError: 330745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13210.857780] LNet: Removed LNI 192.168.202.146@tcp [13211.813763] Key type .llcrypt unregistered [13211.815437] Key type ._llcrypt unregistered [13241.108915] Key type ._llcrypt registered [13241.112872] Key type .llcrypt registered [13241.298678] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [13243.101674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13243.514685] alg: No test for adler32 (adler32-zlib) [13244.765922] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [13245.038834] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [13246.767267] Key type lgssc registered [13249.055781] Lustre: Echo OBD driver; http://www.lustre.org/ [13259.612479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13259.643842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13261.271735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13267.029697] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13271.544237] LustreError: 332025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13271.560024] LustreError: 332025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13276.647591] LustreError: 332026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13278.929367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13279.461094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13285.308123] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13297.015952] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13305.342967] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13314.080580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13314.540988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13321.976345] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13331.798885] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13344.685897] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13346.581994] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13352.428952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13352.439866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13352.455206] Lustre: Skipped 2 previous similar messages [13352.464666] Lustre: Skipped 1 previous similar message [13355.106549] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) [13355.119724] Lustre: Skipped 1 previous similar message [13357.439141] Lustre: server umount lustre-MDT0000 complete [13357.546919] LustreError: 334946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13357.574958] LustreError: 334946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13360.192304] LustreError: 332020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13362.222830] LustreError: 332760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787467503 with bad export cookie 3916992021461273434 [13362.226843] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13362.242900] LustreError: 332760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13362.657951] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13362.693103] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13365.292136] LustreError: 332021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13365.317866] LustreError: 332021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13365.333511] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.46@tcp (stopping) [13368.700600] Lustre: server umount lustre-MDT0001 complete [13373.657861] Lustre: server umount lustre-OST0000 complete [13385.410436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13391.548833] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13395.308196] Lustre: 336211:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.146@tcp [13398.394931] Lustre: server umount MGS complete [13410.448241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13410.968280] LustreError: 336783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13410.995926] LustreError: 336783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13411.092287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13416.638535] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13428.501789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13429.100898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13436.267286] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13447.191880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13447.676499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13454.915532] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13464.092067] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13466.863269] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13473.784884] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13476.272489] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13485.541075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13485.569050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13485.603526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13485.619643] Lustre: Skipped 1 previous similar message [13487.600724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13487.601278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13489.721834] Lustre: server umount lustre-OST0000 complete [13495.777268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13495.788403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13495.803158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13495.824336] Lustre: Skipped 2 previous similar messages [13501.626720] Lustre: server umount lustre-MDT0000 complete [13502.944322] LustreError: 336779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13502.981828] LustreError: 336779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13506.298777] LustreError: 336765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787467647 with bad export cookie 3916992021461274267 [13506.326515] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13506.334617] LustreError: 336765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13507.234137] Lustre: server umount lustre-MDT0001 complete [13526.632329] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13532.216425] Key type lgssc unregistered [13532.530406] LNet: 340250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13532.539278] LNetError: 340250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13532.569048] LNet: Removed LNI 192.168.202.146@tcp [13533.873303] Key type .llcrypt unregistered [13533.876164] Key type ._llcrypt unregistered [13558.298087] Key type ._llcrypt registered [13558.300615] Key type .llcrypt registered [13558.547051] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [13579.498582] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [13580.885038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13580.921715] alg: No test for adler32 (adler32-zlib) [13582.541819] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [13582.825342] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [13584.527183] Key type lgssc registered [13586.171722] Lustre: Echo OBD driver; http://www.lustre.org/ [13636.040194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13636.070460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13637.533802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13637.587608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13637.707866] Lustre: lustre-MDT0000: new disk, initializing [13637.860658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13637.877313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13643.239303] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13654.538697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13654.691922] Lustre: 343417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13654.778396] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13654.782407] Lustre: Skipped 1 previous similar message [13654.859223] Lustre: lustre-MDT0001: new disk, initializing [13654.954252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13655.045060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13655.067985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13660.353624] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13672.435633] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13679.526285] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13686.953518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13687.187565] Lustre: lustre-OST0000: new disk, initializing [13687.193626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13687.206770] Lustre: 345386:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13687.348304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13694.282405] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13696.601633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13696.621290] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13696.765708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13707.202353] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13714.276195] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13714.555991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13720.343747] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13720.677739] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13722.592305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13722.611688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13722.620733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13722.639088] Lustre: Skipped 1 previous similar message [13722.664457] Lustre: Skipped 1 previous similar message [13727.714244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13727.731900] Lustre: Skipped 1 previous similar message [13728.638565] Lustre: server umount lustre-OST0000 complete [13736.929242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13736.939146] LustreError: Skipped 1 previous similar message [13736.945228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13736.973555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13737.954170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13740.151244] Lustre: server umount lustre-MDT0000 complete [13745.194140] LustreError: 343409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787467886 with bad export cookie 15399890529518312099 [13745.222246] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13745.733995] Lustre: server umount lustre-MDT0001 complete [13759.536389] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:51:39 (1787467899) [13781.725531] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [13794.160767] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [13852.246648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13852.588451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13852.652397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13852.809116] Lustre: lustre-MDT0000: new disk, initializing [13853.045524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13853.074499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13859.013685] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13871.851940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13872.006935] Lustre: 350152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13872.085328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13872.089171] Lustre: Skipped 1 previous similar message [13872.170414] Lustre: lustre-MDT0001: new disk, initializing [13872.337178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13872.395559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13872.410650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13878.535914] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13892.452267] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13902.169228] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13910.886741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13911.249643] Lustre: lustre-OST0000: new disk, initializing [13911.257025] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13911.263915] Lustre: 352121:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13911.401898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13918.378699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13918.392933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13918.582745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13921.836642] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13933.650466] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13985.262533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13985.264177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13985.268396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13985.268408] Lustre: Skipped 2 previous similar messages [13985.287937] LustreError: Skipped 1 previous similar message [13985.349939] Lustre: Skipped 1 previous similar message [13988.460710] Lustre: server umount lustre-OST0000 complete [13995.489871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13995.490850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13995.510371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13995.513186] Lustre: Skipped 1 previous similar message [13995.544550] Lustre: Skipped 2 previous similar messages [13999.487419] Lustre: server umount lustre-MDT0000 complete [14000.618456] LustreError: 350159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14000.633101] LustreError: 350159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14004.006583] LustreError: 350143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787468145 with bad export cookie 15399890529518312981 [14004.033291] LustreError: 350143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14004.057763] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14004.404953] Lustre: server umount lustre-MDT0001 complete [14019.953700] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14023.004505] Key type lgssc unregistered [14023.433705] LNet: 354301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14023.442662] LNetError: 354301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14023.469821] LNet: Removed LNI 192.168.202.146@tcp [14024.562069] Key type .llcrypt unregistered [14024.563952] Key type ._llcrypt unregistered [14152.350779] Key type ._llcrypt registered [14152.355292] Key type .llcrypt registered [14152.453455] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14152.988839] Key type .llcrypt unregistered [14152.992900] Key type ._llcrypt unregistered [14250.975960] Key type ._llcrypt registered [14250.982799] Key type .llcrypt registered [14251.202309] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14251.714413] Key type .llcrypt unregistered [14251.718583] Key type ._llcrypt unregistered [14292.175861] Key type ._llcrypt registered [14292.177369] Key type .llcrypt registered [14292.379848] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14292.729456] Key type .llcrypt unregistered [14292.733541] Key type ._llcrypt unregistered [14330.165356] Key type ._llcrypt registered [14330.166879] Key type .llcrypt registered [14330.289648] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14330.728531] Key type .llcrypt unregistered [14330.731539] Key type ._llcrypt unregistered [14433.456392] Key type ._llcrypt registered [14433.462166] Key type .llcrypt registered [14433.673462] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14434.182570] Key type .llcrypt unregistered [14434.185777] Key type ._llcrypt unregistered [14473.752868] Key type ._llcrypt registered [14473.756339] Key type .llcrypt registered [14473.959164] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14474.473586] Key type .llcrypt unregistered [14474.478076] Key type ._llcrypt unregistered [14511.895901] Key type ._llcrypt registered [14511.906494] Key type .llcrypt registered [14512.076588] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14512.508877] Key type .llcrypt unregistered [14512.513108] Key type ._llcrypt unregistered [14555.507591] Key type ._llcrypt registered [14555.510366] Key type .llcrypt registered [14555.841617] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14556.770535] Key type .llcrypt unregistered [14556.772270] Key type ._llcrypt unregistered [14598.977890] Key type ._llcrypt registered [14598.979340] Key type .llcrypt registered [14599.100963] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14599.371957] Key type .llcrypt unregistered [14599.379911] Key type ._llcrypt unregistered [14631.491600] Key type ._llcrypt registered [14631.494110] Key type .llcrypt registered [14631.619753] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14631.957814] Key type .llcrypt unregistered [14631.960663] Key type ._llcrypt unregistered [14710.797753] Key type ._llcrypt registered [14710.799629] Key type .llcrypt registered [14710.905874] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14711.340424] Key type .llcrypt unregistered [14711.346316] Key type ._llcrypt unregistered [14745.965173] Key type ._llcrypt registered [14745.966567] Key type .llcrypt registered [14746.027775] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14746.209462] Key type .llcrypt unregistered [14746.210857] Key type ._llcrypt unregistered [14789.645273] Key type ._llcrypt registered [14789.662274] Key type .llcrypt registered [14789.836915] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14790.421409] Key type .llcrypt unregistered [14790.423385] Key type ._llcrypt unregistered [14860.151372] Key type ._llcrypt registered [14860.154945] Key type .llcrypt registered [14860.391048] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [14879.781820] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [14880.616877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14880.656558] alg: No test for adler32 (adler32-zlib) [14881.639458] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [14881.846776] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [14883.511827] Key type lgssc registered [14885.311302] Lustre: Echo OBD driver; http://www.lustre.org/ [14934.532517] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14934.564057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14935.885674] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14935.925183] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14936.091635] Lustre: tst250fs-MDT0000: new disk, initializing [14936.322379] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14936.430634] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14943.894836] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [14956.229470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14956.418511] Lustre: 374729:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14956.477315] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14956.484406] Lustre: Skipped 1 previous similar message [14956.620308] Lustre: tst250fs-MDT0001: new disk, initializing [14956.751131] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14956.817281] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14956.831157] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14962.961258] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [14969.105784] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14978.035747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14978.733531] Lustre: tst250fs-OST0000: new disk, initializing [14978.737950] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14978.744821] Lustre: 376361:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14978.912421] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14979.855765] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14979.895669] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14980.166986] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14986.807094] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [14998.007725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14998.258306] Lustre: tst250fs-OST0001: new disk, initializing [14998.269760] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14998.286885] Lustre: 377234:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14998.412376] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [15004.361928] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [15008.416974] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [15008.429758] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [15008.562608] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15010.341863] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [15012.343698] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [15012.352189] Lustre: Skipped 1 previous similar message [15038.944678] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15038.951923] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15038.972153] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [15039.462065] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15039.498445] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [15039.514535] Lustre: Skipped 3 previous similar messages [15044.579779] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [15044.587062] Lustre: Skipped 2 previous similar messages [15049.699984] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [15049.717916] Lustre: Skipped 1 previous similar message [15050.208685] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15052.474178] Lustre: server umount tst250fs-MDT0000 complete [15058.146152] LustreError: 374723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787469199 with bad export cookie 11420624807948472053 [15058.159171] LustreError: 374723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15058.165189] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15058.797316] Lustre: server umount tst250fs-MDT0001 complete [15074.353258] Lustre: server umount tst250fs-OST0000 complete [15076.322477] Lustre: 372952:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787469201/real 1787469201] req@ffff981250868a80 x1874297117286784/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787469217 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15076.346088] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15076.366085] Lustre: Skipped 2 previous similar messages [15078.524267] Lustre: server umount tst250fs-OST0001 complete [15103.136291] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [15117.408374] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [15156.159818] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 03:14:56 (1787469296) [15158.626910] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [15178.438551] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [15194.975243] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [15236.924146] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14929 sec ========= 03:16:16 (1787469376) [15239.385746] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:16:19 (1787469379) === [15244.069489] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:16:24 (1787469384) === [15298.530433] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [15302.892113] Key type lgssc unregistered [15303.340779] LNet: 386416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15303.359021] LNetError: 386416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15303.381561] LNet: Removed LNI 192.168.202.146@tcp [15304.854140] Key type .llcrypt unregistered [15304.856558] Key type ._llcrypt unregistered