[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 498741233 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002391] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007675] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010186] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013073] *** VALIDATE selinux *** [ 0.021764] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026285] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029021] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030107] *** VALIDATE tmpfs *** [ 0.032102] *** VALIDATE proc *** [ 0.033243] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.036125] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039034] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.042447] debug: unmapping init [mem 0xffffffff9ca59000-0xffffffff9ca60fff] [ 0.044181] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045747] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052015] ... event mask: 000000070000000f [ 0.053297] rcu: Hierarchical SRCU implementation. [ 0.055407] smp: Bringing up secondary CPUs ... [ 0.056591] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.060448] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.130514] node 0 deferred pages initialised in 64ms [ 0.132355] devtmpfs: initialized [ 0.133282] x86/mm: Memory block size: 128MB [ 0.136028] gcov: version magic: 0x41383552 [ 0.138363] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139107] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.140223] pinctrl core: initialized pinctrl subsystem [ 0.141210] [ 0.141714] ************************************************************* [ 0.142023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143021] ** ** [ 0.144014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.145013] ** ** [ 0.146012] ** This means that this kernel is built to expose internal ** [ 0.147015] ** IOMMU data structures, which may compromise security on ** [ 0.148014] ** your system. ** [ 0.149015] ** ** [ 0.150015] ** If you see this message and you are not debugging the ** [ 0.151011] ** kernel, report this immediately to your vendor! ** [ 0.152016] ** ** [ 0.153017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154016] ************************************************************* [ 0.155483] NET: Registered protocol family 16 [ 0.156434] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.157077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.158074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.159485] cpuidle: using governor menu [ 0.162178] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.164748] PCI: Using configuration type 1 for base access [ 0.165160] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.175133] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.176037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.178067] cryptd: max_cpu_qlen set to 1000 [ 0.180292] ACPI: Added _OSI(Module Device) [ 0.181044] ACPI: Added _OSI(Processor Device) [ 0.182016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.183015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.187220] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.194421] ACPI: Interpreter enabled [ 0.196067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.197012] ACPI: Using IOAPIC for interrupt routing [ 0.199139] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.202426] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.214664] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.216050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.219024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.223080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.227615] acpiphp: Slot [2] registered [ 0.229154] acpiphp: Slot [5] registered [ 0.230156] acpiphp: Slot [6] registered [ 0.232120] acpiphp: Slot [7] registered [ 0.233115] acpiphp: Slot [8] registered [ 0.234136] acpiphp: Slot [9] registered [ 0.235126] acpiphp: Slot [10] registered [ 0.237109] acpiphp: Slot [3] registered [ 0.238134] acpiphp: Slot [4] registered [ 0.240094] acpiphp: Slot [11] registered [ 0.241109] acpiphp: Slot [12] registered [ 0.243133] acpiphp: Slot [13] registered [ 0.244084] acpiphp: Slot [14] registered [ 0.245082] acpiphp: Slot [15] registered [ 0.246100] acpiphp: Slot [16] registered [ 0.248101] acpiphp: Slot [17] registered [ 0.249083] acpiphp: Slot [18] registered [ 0.251083] acpiphp: Slot [19] registered [ 0.252126] acpiphp: Slot [20] registered [ 0.253111] acpiphp: Slot [21] registered [ 0.255126] acpiphp: Slot [22] registered [ 0.257152] acpiphp: Slot [23] registered [ 0.259164] acpiphp: Slot [24] registered [ 0.260213] acpiphp: Slot [25] registered [ 0.262201] acpiphp: Slot [26] registered [ 0.264155] acpiphp: Slot [27] registered [ 0.267191] acpiphp: Slot [28] registered [ 0.268136] acpiphp: Slot [29] registered [ 0.270175] acpiphp: Slot [30] registered [ 0.272139] acpiphp: Slot [31] registered [ 0.273127] PCI host bridge to bus 0000:00 [ 0.275041] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.277038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.280036] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.283038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.286038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.289053] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.291347] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.296678] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.301031] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.315660] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.321762] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.324038] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.329039] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.332029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.335131] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.339210] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.341055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344887] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.351023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.363019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.369033] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.376379] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.386024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.399019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.420027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.428689] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.437026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.444024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.462027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.472971] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.478019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.483020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.499035] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.509175] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.515000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.519022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.535034] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.545700] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.555024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.563019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.578026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.591032] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.597017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.603019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.621024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.631239] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.633584] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.637548] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.640520] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.643324] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.649271] iommu: Default domain type: Passthrough [ 0.651467] SCSI subsystem initialized [ 0.652195] ACPI: bus type USB registered [ 0.654129] usbcore: registered new interface driver usbfs [ 0.656097] usbcore: registered new interface driver hub [ 0.658085] usbcore: registered new device driver usb [ 0.659187] pps_core: LinuxPPS API ver. 1 registered [ 0.661021] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.664090] PTP clock support registered [ 0.666229] EDAC MC: Ver: 3.0.0 [ 0.669231] PCI: Using ACPI for IRQ routing [ 0.671039] NetLabel: Initializing [ 0.673013] NetLabel: domain hash size = 128 [ 0.674013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.676154] NetLabel: unlabeled traffic allowed by default [ 0.679114] vgaarb: loaded [ 0.681050] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.682016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.688568] clocksource: Switched to clocksource kvm-clock [ 0.798655] VFS: Disk quotas dquot_6.6.0 [ 0.800068] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.802915] *** VALIDATE ramfs *** [ 0.804347] *** VALIDATE hugetlbfs *** [ 0.806195] pnp: PnP ACPI init [ 0.808729] pnp: PnP ACPI: found 6 devices [ 0.829321] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.832273] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.834158] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.835752] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.838135] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.840929] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.843382] NET: Registered protocol family 2 [ 0.845962] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.851395] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.856114] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.861782] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.865251] TCP: Hash tables configured (established 65536 bind 65536) [ 0.867841] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.870693] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.873532] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.876483] NET: Registered protocol family 1 [ 0.879502] RPC: Registered named UNIX socket transport module. [ 0.881966] RPC: Registered udp transport module. [ 0.884150] RPC: Registered tcp transport module. [ 0.886319] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.889049] NET: Registered protocol family 44 [ 0.891080] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.893784] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.896298] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.898982] PCI: CLS 0 bytes, default 64 [ 0.901363] Unpacking initramfs... [ 2.368581] debug: unmapping init [mem 0xffff9a21bcc54000-0xffff9a21bffbffff] [ 2.373129] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.375976] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.381277] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 2.887573] Initialise system trusted keyrings [ 2.889947] Key type blacklist registered [ 2.891864] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.901466] zbud: loaded [ 2.904677] *** VALIDATE nfs *** [ 2.906104] *** VALIDATE nfs4 *** [ 2.908239] pstore: using deflate compression [ 2.914559] Platform Keyring initialized [ 3.017549] NET: Registered protocol family 38 [ 3.019571] Key type asymmetric registered [ 3.021337] Asymmetric key parser 'x509' registered [ 3.023799] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.026676] io scheduler mq-deadline registered [ 3.028377] io scheduler kyber registered [ 3.030285] io scheduler bfq registered [ 3.031951] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.035476] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.038142] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.040204] ACPI: Power Button [PWRF] [ 3.046487] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.055572] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.067862] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.074273] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.104456] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.133581] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.162272] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.168727] Non-volatile memory driver v1.3 [ 3.169841] Linux agpgart interface v0.103 [ 3.211867] virtio_blk virtio1: [vda] 134864 512-byte logical blocks (69.1 MB/65.9 MiB) [ 3.214879] vda: detected capacity change from 0 to 69050368 [ 3.229917] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.233622] vdb: detected capacity change from 0 to 1073741824 [ 3.249945] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.254188] vdc: detected capacity change from 0 to 2621440000 [ 3.268243] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.270986] vdd: detected capacity change from 0 to 2621440000 [ 3.286580] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.289501] vde: detected capacity change from 0 to 4294967296 [ 3.304680] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.308347] vdf: detected capacity change from 0 to 4294967296 [ 3.314930] libphy: Fixed MDIO Bus: probed [ 3.322660] usbcore: registered new interface driver usbserial_generic [ 3.325268] usbserial: USB Serial support registered for generic [ 3.328320] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.333139] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.335262] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.338368] mousedev: PS/2 mouse device common for all mice [ 3.343914] rtc_cmos 00:05: RTC can wake from S4 [ 3.347305] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.350582] rtc_cmos 00:05: registered as rtc0 [ 3.354235] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.356961] intel_pstate: CPU model not supported [ 3.359752] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.361375] hid: raw HID events driver (C) Jiri Kosina [ 3.365381] usbcore: registered new interface driver usbhid [ 3.367463] usbhid: USB HID core driver [ 3.369539] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.369564] drop_monitor: Initializing network drop monitor service [ 3.377299] Initializing XFRM netlink socket [ 3.379433] NET: Registered protocol family 10 [ 3.382571] Segment Routing with IPv6 [ 3.383862] NET: Registered protocol family 17 [ 3.386160] mpls_gso: MPLS GSO support [ 3.391831] RAS: Correctable Errors collector initialized. [ 3.393625] AVX version of gcm_enc/dec engaged. [ 3.395252] AES CTR mode by8 optimization enabled [ 3.477759] sched_clock: Marking stable (3477732871, 0)->(4465180341, -987447470) [ 3.481783] registered taskstats version 1 [ 3.484023] Loading compiled-in X.509 certificates [ 3.486381] zswap: loaded using pool lzo/zbud [ 3.510694] Key type big_key registered [ 3.524522] Key type encrypted registered [ 3.526415] ima: No TPM chip found, activating TPM-bypass! [ 3.528710] ima: Allocated hash algorithm: sha1 [ 3.530632] ima: No architecture policies found [ 3.532575] evm: Initialising EVM extended attributes: [ 3.534598] evm: security.selinux [ 3.535895] evm: security.ima [ 3.537212] evm: security.capability [ 3.538618] evm: HMAC attrs: 0x1 [ 3.541279] rtc_cmos 00:05: setting system clock to 2026-06-01 17:59:42 UTC (1780336782) [ 3.547991] debug: unmapping init [mem 0xffffffff9da03000-0xffffffff9dbfffff] [ 3.551450] debug: unmapping init [mem 0xffffffff9c782000-0xffffffff9ca58fff] [ 3.562148] Write protecting the kernel read-only data: 28672k [ 3.565627] debug: unmapping init [mem 0xffffffff9ae03000-0xffffffff9affffff] [ 3.568961] debug: unmapping init [mem 0xffffffff9b714000-0xffffffff9b7fffff] [ 3.605831] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.614454] systemd[1]: Detected virtualization kvm. [ 3.616546] systemd[1]: Detected architecture x86-64. [ 3.618414] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.644892] systemd[1]: No hostname configured. [ 3.647072] systemd[1]: Set hostname to . [ 3.649383] random: systemd: uninitialized urandom read (16 bytes read) [ 3.652388] systemd[1]: Initializing machine ID from random generator. [ 3.712394] random: ln: uninitialized urandom read (6 bytes read) [ 3.811877] random: systemd: uninitialized urandom read (16 bytes read) [ 3.813979] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.818197] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.823363] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.519401] device-mapper: uevent: version 1.0.3 [ 4.521329] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK [ 5.225583] random: fast init done ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.297255] virtio_net virtio0 ens2: renamed from eth0 [ 5.401320] scsi host0: ata_piix [ 5.412618] scsi host1: ata_piix [ 5.414582] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.417250] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.725865] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.046731] random: crng init done [ 10.048542] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.531835] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.724055] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.027105] SELinux: Disabled at runtime. [ 12.088981] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.097673] systemd[1]: Detected virtualization kvm. [ 12.099463] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.633403] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.637456] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.643302] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.646650] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.650317] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.658211] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.667501] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ 12.845736] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.175600] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.539817] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.635874] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.749546] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.786380] EDAC sbridge: Ver: 1.1.2 [ 15.640302] Key type dns_resolver registered [ 15.954708] NFS: Registering the id_resolver key type [ 15.956949] Key type id_resolver registered [ 15.958769] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg459-server login: [ 86.260386] libcfs: loading out-of-tree module taints kernel. [ 86.298227] Key type ._llcrypt registered [ 86.300421] Key type .llcrypt registered [ 86.414611] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 105.161539] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 107.685666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 107.708618] alg: No test for adler32 (adler32-zlib) [ 109.463296] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 110.442951] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 112.153116] Key type lgssc registered [ 112.360127] hrtimer: interrupt took 4117124 ns [ 113.539851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 130.422975] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 178.335162] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 191.386425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 191.425288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.925161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.014057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.184077] Lustre: lustre-MDT0000: new disk, initializing [ 193.319698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.346439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 197.568482] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.850679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.943813] Lustre: 6524:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 210.987932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 211.002837] Lustre: Skipped 1 previous similar message [ 211.165706] Lustre: lustre-MDT0001: new disk, initializing [ 211.287591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.332082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 211.346946] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.477871] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.858037] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 233.768918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.005559] Lustre: lustre-OST0000: new disk, initializing [ 234.010824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.018612] Lustre: 8432:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.082669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 239.698121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 239.715890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 239.824289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 240.598517] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.128571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 257.292901] Lustre: lustre-OST0001: new disk, initializing [ 257.296935] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 257.303042] Lustre: 9487:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 257.412495] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 264.595893] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 267.300414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 267.318872] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 267.365371] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 277.410540] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 288.900479] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 294.929884] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing check_logdir /tmp/testlogs/ [ 300.240169] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing yml_node [ 304.304214] Lustre: DEBUG MARKER: Client: 2.17.53.34 [ 306.984541] Lustre: DEBUG MARKER: MDS: 2.17.53.34 [ 310.319711] Lustre: DEBUG MARKER: OSS: 2.17.53.34 [ 312.143442] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jun 1 14:04:50 EDT 2026 [ 332.547565] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 334.584556] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 354.278349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.280883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.300925] Lustre: Skipped 3 previous similar messages [ 354.319438] Lustre: Skipped 3 previous similar messages [ 358.027832] Lustre: server umount lustre-MDT0000 complete [ 361.822700] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337140 with bad export cookie 5079103645164390950 [ 361.830832] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 361.835782] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 364.513122] LustreError: 6531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.513867] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.514326] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 364.537987] LustreError: 6531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 364.574797] Lustre: Skipped 1 previous similar message [ 366.563701] LustreError: 6531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.569813] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.594190] Lustre: Skipped 2 previous similar messages [ 371.698224] LustreError: 6531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.703804] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.734592] LustreError: 6531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 371.755921] Lustre: Skipped 1 previous similar message [ 376.288157] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 376.641088] Lustre: server umount lustre-MDT0001 complete [ 381.511683] Lustre: server umount lustre-OST0000 complete [ 386.716137] Lustre: server umount lustre-OST0001 complete [ 394.831467] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 404.846514] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 441.539921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.020944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 442.067147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 442.210562] Lustre: lustre-MDT0000: new disk, initializing [ 442.302462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.329401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 447.301293] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 459.903968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.031239] Lustre: 15009:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 460.119756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 460.131217] Lustre: Skipped 1 previous similar message [ 460.301528] Lustre: lustre-MDT0001: new disk, initializing [ 460.419961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 460.492062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 460.502457] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 465.586356] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 474.283639] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 478.879651] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 486.625565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 486.921965] Lustre: lustre-OST0000: new disk, initializing [ 486.926382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 486.932564] Lustre: 16756:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 487.014771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 488.256202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 488.271877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 488.346553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 492.943150] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 500.670586] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 506.183827] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 506.478535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 511.251020] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 511.499590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 514.018436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 514.037916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.042268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 514.064918] Lustre: Skipped 1 previous similar message [ 514.076306] Lustre: Skipped 1 previous similar message [ 524.260115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 524.274362] Lustre: Skipped 3 previous similar messages [ 527.842667] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 528.120696] Lustre: server umount lustre-OST0000 complete [ 534.498325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.540930] Lustre: Skipped 1 previous similar message [ 539.619033] LustreError: 15016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.644966] LustreError: 15016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 540.846627] Lustre: server umount lustre-MDT0000 complete [ 544.741366] LustreError: 15021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.773516] LustreError: 15021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 545.178689] LustreError: 15003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337324 with bad export cookie 5079103645164392518 [ 545.189619] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 545.193026] LustreError: 15003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 545.542576] Lustre: server umount lustre-MDT0001 complete [ 552.485970] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:08:49 (1780337329) [ 573.060171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.647055] LustreError: 19485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.760880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 578.768226] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 588.890703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.410623] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 602.400901] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 606.908832] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 609.219624] Lustre: Failing over lustre-MDT0000 [ 609.762907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 609.763707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.770627] LustreError: Skipped 1 previous similar message [ 609.792122] LustreError: 19482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.792134] LustreError: 19482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 611.582747] Lustre: server umount lustre-MDT0000 complete [ 630.247105] Lustre: 3675:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780337393/real 1780337393] req@ffff9a22349a4000 x1866818533896064/t0(0) o400->MGC192.168.204.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780337409 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 630.295733] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 632.718923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.481899] LustreError: 3673:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9a223d189c00 x1866818533898752/t0(0) o250->MGC192.168.204.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 640.910927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 640.919564] Lustre: Skipped 1 previous similar message [ 641.012571] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 646.132301] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 646.150506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 646.164423] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 646.485426] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 656.448268] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 661.476967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 661.487205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.518608] Lustre: Skipped 1 previous similar message [ 661.529585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 661.533232] Lustre: Skipped 2 previous similar messages [ 667.617942] LustreError: 20211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 667.634388] LustreError: 20211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 669.047208] Lustre: server umount lustre-MDT0000 complete [ 673.475117] LustreError: 21815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337452 with bad export cookie 5079103645164393981 [ 673.478727] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 673.484055] LustreError: 21815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 673.965661] Lustre: server umount lustre-MDT0001 complete [ 683.437474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 688.932324] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 698.965285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.954395] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 711.476710] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 715.553408] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 717.876487] Lustre: Failing over lustre-MDT0000 [ 718.133470] Lustre: server umount lustre-MDT0000 complete [ 719.329253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 719.339696] Lustre: Skipped 1 previous similar message [ 734.691467] LustreError: 22486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 734.714026] LustreError: 22486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 735.526756] Lustre: 3676:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780337498/real 1780337498] req@ffff9a223b442a00 x1866818533926144/t0(0) o400->MGC192.168.204.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780337514 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 735.561571] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 737.983964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 746.535946] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 751.207540] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 751.591691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 751.592956] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 751.598331] Lustre: Skipped 1 previous similar message [ 751.640788] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 759.019491] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 763.140865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 763.150681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 763.164831] Lustre: Skipped 2 previous similar messages [ 763.187339] Lustre: Skipped 3 previous similar messages [ 769.469712] Lustre: server umount lustre-MDT0000 complete [ 774.133698] LustreError: 23211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337553 with bad export cookie 5079103645164394919 [ 774.136987] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 774.152674] LustreError: 23211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 774.604959] Lustre: server umount lustre-MDT0001 complete [ 783.930575] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:12:41 (1780337561) [ 800.493609] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 812.410863] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 853.515076] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 865.792589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.162786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 866.200462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 866.298256] Lustre: lustre-MDT0000: new disk, initializing [ 866.455667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.466861] Lustre: Skipped 3 previous similar messages [ 866.498819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 871.401879] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 884.679571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.920324] Lustre: 28495:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 885.392905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 891.263556] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 902.684156] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 908.491796] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 916.520233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 916.850643] Lustre: 30241:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 918.830923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 918.882762] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 918.957416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 918.970667] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 923.698164] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 931.247141] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 937.134487] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 959.969406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 959.983765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 959.992897] Lustre: Skipped 1 previous similar message [ 959.994672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 960.005148] Lustre: Skipped 3 previous similar messages [ 974.304175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 974.562487] Lustre: server umount lustre-OST0000 complete [ 984.760735] Lustre: server umount lustre-MDT0000 complete [ 985.570452] LustreError: 29341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 985.596216] LustreError: 29341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 988.780532] LustreError: 29274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337767 with bad export cookie 5079103645164395318 [ 988.783366] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 988.807569] LustreError: 29274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 989.467323] Lustre: server umount lustre-MDT0001 complete [ 1001.508114] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1004.949556] Key type lgssc unregistered [ 1005.282410] LNet: 32092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.293299] LNetError: 32092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.323865] LNet: Removed LNI 192.168.204.159@tcp [ 1007.603196] Key type .llcrypt unregistered [ 1007.606807] Key type ._llcrypt unregistered [ 1036.291410] Key type ._llcrypt registered [ 1036.293240] Key type .llcrypt registered [ 1036.469619] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1037.373319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1037.398220] alg: No test for adler32 (adler32-zlib) [ 1038.570345] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 1038.870621] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1040.568192] Key type lgssc registered [ 1042.001237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1050.945564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1050.973719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.689489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.678961] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1062.892266] LustreError: 33488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1062.918291] LustreError: 33488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1067.100872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.569764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1072.304593] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1078.387392] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1082.085518] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1088.152988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1088.478442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.624779] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1093.822233] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1097.799856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1099.891575] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1113.706446] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:18:11 (1780337891) [ 1115.658517] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1117.665427] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1125.274994] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:18:23 (1780337903) [ 1133.922801] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:18:32 (1780337912) [ 1134.948452] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1143.942558] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1143.948204] Lustre: Skipped 2 previous similar messages [ 1145.994902] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:18:43 (1780337923) [ 1147.986708] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1147.991339] Lustre: Skipped 2 previous similar messages [ 1167.066241] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1167.071708] Lustre: Skipped 7 previous similar messages [ 1169.731265] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:19:07 (1780337947) [ 1172.497596] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1176.056583] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1185.314464] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:19:23 (1780337963) [ 1187.380841] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1187.385383] Lustre: Skipped 3 previous similar messages [ 1200.733907] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:19:38 (1780337978) [ 1215.458645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1215.472517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.493831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1215.502728] Lustre: Skipped 1 previous similar message [ 1220.585511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1220.593735] Lustre: Skipped 1 previous similar message [ 1225.699136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1225.712631] Lustre: Skipped 1 previous similar message [ 1229.280118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1229.422672] Lustre: server umount lustre-OST0000 complete [ 1235.938209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.949314] Lustre: Skipped 1 previous similar message [ 1235.953822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1235.956975] Lustre: Skipped 1 previous similar message [ 1241.058560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.071555] Lustre: Skipped 1 previous similar message [ 1247.201205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1247.409230] Lustre: server umount lustre-MDT0000 complete [ 1250.669354] LustreError: 33471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338029 with bad export cookie 6872676570884509595 [ 1250.682188] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1250.685679] LustreError: 33471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1251.009301] Lustre: server umount lustre-MDT0001 complete [ 1261.205134] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1264.008466] Key type lgssc unregistered [ 1264.385256] LNet: 40993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.395728] LNetError: 40993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.420270] LNet: Removed LNI 192.168.204.159@tcp [ 1265.311517] Key type .llcrypt unregistered [ 1265.314048] Key type ._llcrypt unregistered [ 1282.511556] Key type ._llcrypt registered [ 1282.513774] Key type .llcrypt registered [ 1282.600573] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 1298.764341] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1299.608120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1299.617837] alg: No test for adler32 (adler32-zlib) [ 1300.681439] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 1300.926440] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1302.656239] Key type lgssc registered [ 1304.275348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1346.150726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1346.199426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1347.493325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1347.529949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1347.649575] Lustre: lustre-MDT0000: new disk, initializing [ 1347.817764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1347.838900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1352.254098] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1363.481155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.619283] Lustre: 44114:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1363.722919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1363.736708] Lustre: Skipped 1 previous similar message [ 1363.833827] Lustre: lustre-MDT0001: new disk, initializing [ 1363.972479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1364.035339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1364.040744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1369.491991] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1378.043952] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1382.409308] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1389.430523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1389.676204] Lustre: lustre-OST0000: new disk, initializing [ 1389.680693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1389.687881] Lustre: 45860:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1389.786855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.581422] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1398.874781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1398.884466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1399.048070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1405.163517] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1409.879400] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1410.122670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1414.123885] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1414.318963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1419.750051] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1419.751982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.754535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1419.768398] LustreError: Skipped 1 previous similar message [ 1419.794957] Lustre: Skipped 1 previous similar message [ 1422.430513] Lustre: server umount lustre-OST0000 complete [ 1429.986974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.997809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1430.002840] Lustre: Skipped 2 previous similar messages [ 1432.898609] Lustre: server umount lustre-MDT0000 complete [ 1436.274855] LustreError: 44107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338215 with bad export cookie 2631546558949211257 [ 1436.278924] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1436.285439] LustreError: 44107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1436.629758] Lustre: server umount lustre-MDT0001 complete [ 1447.057750] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:23:44 (1780338224) [ 1454.768346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1455.150736] LustreError: 48182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.175541] LustreError: 48182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1455.213542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.686597] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1460.723602] LustreError: 48183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1465.827186] LustreError: 48182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1469.956709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.564958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1475.745712] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1483.926591] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1488.335494] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1496.080101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1496.471625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1503.105049] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1509.721586] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1517.252615] LustreError: 50376:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1518.059484] Lustre: Modifying parameter general.timeout=1 in log params [ 1519.725657] Lustre: Modifying parameter general.timeout=3 in log params [ 1519.729065] Lustre: Skipped 1 previous similar message [ 1522.214429] Lustre: Modifying parameter general.timeout=6 in log params [ 1522.218467] Lustre: Skipped 2 previous similar messages [ 1526.669690] Lustre: Modifying parameter general.timeout=11 in log params [ 1526.679683] Lustre: Skipped 4 previous similar messages [ 1535.209187] Lustre: Modifying parameter general.timeout=20 in log params [ 1535.212227] Lustre: Skipped 8 previous similar messages [ 1551.555438] Lustre: Modifying parameter general.timeout=32 in log params [ 1551.561199] Lustre: Skipped 11 previous similar messages [ 1584.099884] Lustre: Modifying parameter general.timeout=66 in log params [ 1584.114864] Lustre: Skipped 33 previous similar messages [ 1648.768337] Lustre: Modifying parameter general.timeout=125 in log params [ 1648.777151] Lustre: Skipped 58 previous similar messages [ 1664.625701] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:27:22 (1780338442) [ 1667.841994] Lustre: 57325:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1669.931429] Lustre: 57423:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1672.305821] Lustre: 57535:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1678.110463] Lustre: 57844:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1678.118270] Lustre: 57844:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1706.090732] Lustre: 59206:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1706.114963] LustreError: 59206:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1709.895612] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:28:07 (1780338487) [ 1722.340651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.348081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1722.357705] Lustre: Skipped 3 previous similar messages [ 1722.364496] Lustre: Skipped 1 previous similar message [ 1727.465086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1727.478660] Lustre: Skipped 2 previous similar messages [ 1727.995331] Lustre: server umount lustre-MDT0000 complete [ 1732.159762] LustreError: 48163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338511 with bad export cookie 2631546558949212181 [ 1732.172568] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1732.175060] LustreError: 48163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1732.583736] Lustre: server umount lustre-MDT0001 complete [ 1747.011550] Lustre: server umount lustre-OST0000 complete [ 1775.615255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1775.911125] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1776.187532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1781.700623] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1792.856320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.217691] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1793.290162] Lustre: 61736:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1799.184771] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1805.171787] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1809.376986] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1816.849966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1817.107984] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1817.418103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.427117] Lustre: Skipped 1 previous similar message [ 1823.655864] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1829.448294] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1831.099469] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1831.120545] Lustre: Skipped 28 previous similar messages [ 1831.176479] Lustre: 63966:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1840.609318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1840.616901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.642576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1845.345621] Lustre: server umount lustre-OST0000 complete [ 1851.362175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.392913] Lustre: Skipped 1 previous similar message [ 1851.410404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.431712] Lustre: Skipped 3 previous similar messages [ 1857.147384] Lustre: server umount lustre-MDT0000 complete [ 1861.168433] LustreError: 61726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338640 with bad export cookie 2631546558949215345 [ 1861.178037] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1861.190744] LustreError: 61726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1861.790677] Lustre: server umount lustre-MDT0001 complete [ 1874.619506] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1877.264573] Key type lgssc unregistered [ 1877.510794] LNet: 65161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.519891] LNetError: 65161:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.545380] LNet: Removed LNI 192.168.204.159@tcp [ 1878.310847] Key type .llcrypt unregistered [ 1878.313100] Key type ._llcrypt unregistered [ 1907.051943] Key type ._llcrypt registered [ 1907.060560] Key type .llcrypt registered [ 1907.312347] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1908.700192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1908.717456] alg: No test for adler32 (adler32-zlib) [ 1909.803920] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 1910.099780] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1911.848197] Key type lgssc registered [ 1913.102858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1921.878625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1921.911509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.703513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.567903] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1933.800549] LustreError: 66559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1933.830774] LustreError: 66559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1937.715882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.412442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1942.958539] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1949.795394] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1954.933468] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1961.509602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1962.009164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.296382] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1974.644931] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1983.667269] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1993.711951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.714123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1993.734869] Lustre: Skipped 1 previous similar message [ 1993.765867] Lustre: Skipped 2 previous similar messages [ 1998.821912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1998.831891] Lustre: Skipped 2 previous similar messages [ 2000.175161] Lustre: server umount lustre-MDT0000 complete [ 2003.939071] LustreError: 67275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2004.692832] LustreError: 66539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338783 with bad export cookie 2676092068156917570 [ 2004.697056] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.701575] LustreError: 66539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2005.190761] Lustre: server umount lustre-MDT0001 complete [ 2019.476666] Lustre: server umount lustre-OST0000 complete [ 2052.804297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.194702] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2053.221460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2053.585552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2058.429658] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2067.834936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.901630] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2067.930143] Lustre: 71095:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2068.364675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2073.421993] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2079.992272] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2084.128486] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2092.000421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2092.249748] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2092.656657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.517816] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2106.695325] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2108.767891] Lustre: 73333:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2110.715966] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2120.690490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2120.695582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.711488] LustreError: Skipped 1 previous similar message [ 2120.717995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2120.725145] Lustre: Skipped 2 previous similar messages [ 2120.746032] Lustre: Skipped 1 previous similar message [ 2123.339182] Lustre: server umount lustre-OST0000 complete [ 2129.889366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2129.896588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.910199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2133.729316] Lustre: server umount lustre-MDT0000 complete [ 2136.035623] LustreError: 71101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.047289] LustreError: 71101:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2137.664136] LustreError: 72737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338916 with bad export cookie 2676092068156918417 [ 2137.666389] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2137.694769] LustreError: 72737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2138.024208] Lustre: server umount lustre-MDT0001 complete [ 2153.792303] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2158.176571] Key type lgssc unregistered [ 2158.646202] LNet: 74527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.665734] LNetError: 74527:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.707165] LNet: Removed LNI 192.168.204.159@tcp [ 2159.974949] Key type .llcrypt unregistered [ 2159.977384] Key type ._llcrypt unregistered [ 2188.048734] Key type ._llcrypt registered [ 2188.054406] Key type .llcrypt registered [ 2188.184521] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2189.410467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2189.440483] alg: No test for adler32 (adler32-zlib) [ 2190.797741] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 2191.099593] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2192.914111] Key type lgssc registered [ 2194.294433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2202.554898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2202.602639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2204.264108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.574691] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2214.374176] LustreError: 75924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.400891] LustreError: 75924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2219.494454] LustreError: 75925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2219.983290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.618464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2226.005864] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2232.202854] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2236.038123] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2243.142600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2243.869446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.253556] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2256.244796] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2262.210681] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2263.238556] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2264.357663] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2266.545227] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2266.554725] Lustre: Skipped 1 previous similar message [ 2270.613588] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2270.617189] Lustre: Skipped 3 previous similar messages [ 2279.105757] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2279.110778] Lustre: Skipped 8 previous similar messages [ 2295.843295] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2295.850604] Lustre: Skipped 15 previous similar messages [ 2328.503617] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2328.510726] Lustre: Skipped 28 previous similar messages [ 2375.448743] Lustre: 83696:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2378.771887] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 14:39:16 (1780339156) [ 2380.518848] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2382.512748] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 14:39:20 (1780339160) [ 2388.457729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.480462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2388.487199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2388.503404] Lustre: Skipped 1 previous similar message [ 2393.574209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2393.595224] Lustre: Skipped 1 previous similar message [ 2398.696377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2398.708063] Lustre: Skipped 1 previous similar message [ 2401.248102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2401.531644] Lustre: server umount lustre-OST0000 complete [ 2408.942777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2408.953019] Lustre: Skipped 1 previous similar message [ 2408.960967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2412.063343] Lustre: server umount lustre-MDT0000 complete [ 2414.050971] LustreError: 76640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.071123] LustreError: 76640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2416.072972] LustreError: 75905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339195 with bad export cookie 14440685044540482843 [ 2416.075398] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2416.087614] LustreError: 75905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2416.703826] Lustre: server umount lustre-MDT0001 complete [ 2428.705027] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2431.901442] Key type lgssc unregistered [ 2432.215891] LNet: 84932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2432.229641] LNetError: 84932:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2432.248928] LNet: Removed LNI 192.168.204.159@tcp [ 2433.143759] Key type .llcrypt unregistered [ 2433.146913] Key type ._llcrypt unregistered [ 2437.617617] Key type ._llcrypt registered [ 2437.628115] Key type .llcrypt registered [ 2437.761234] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2442.660607] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules [ 2443.918878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2443.959228] alg: No test for adler32 (adler32-zlib) [ 2445.194126] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 2445.425689] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2447.088187] Key type lgssc registered [ 2448.717983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2450.361419] LustreError: 86108:0:(lu_object.c:1479:lu_context_key_register()) cfs_race id 60d sleeping [ 2470.085250] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2472.487185] LustreError: 86108:0:(lu_object.c:1479:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2480.797015] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2480.834481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.692906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.502972] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2492.903899] LustreError: 87106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.931250] LustreError: 87106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2497.144663] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 14:41:14 (1780339274) [ 2498.020038] LustreError: 87105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.992264] LustreError: 87138:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2503.028994] LustreError: 87138:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2503.043961] LustreError: 87138:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2503.694223] Lustre: server umount lustre-MDT0000 complete [ 2520.174477] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2523.747525] Key type lgssc unregistered [ 2524.194704] LNet: 88600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2524.207434] LNetError: 88600:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2524.220244] LNet: Removed LNI 192.168.204.159@tcp [ 2525.165647] Key type .llcrypt unregistered [ 2525.170420] Key type ._llcrypt unregistered [ 2545.542569] Key type ._llcrypt registered [ 2545.545484] Key type .llcrypt registered [ 2545.676598] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2547.470081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2547.531269] alg: No test for adler32 (adler32-zlib) [ 2548.908525] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 2549.246759] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2550.952241] Key type lgssc registered [ 2552.502739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2561.981749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2562.012839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.680471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.677933] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2573.802472] LustreError: 89802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.826318] LustreError: 89802:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2578.387118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.078114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2584.024067] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2591.530597] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2597.500644] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2605.770182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2606.686767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2613.592451] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2619.875891] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2677.290370] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 14:44:15 (1780339455) [ 2679.417948] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2681.949333] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 14:44:19 (1780339459) [ 2686.433358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2686.445668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2686.482528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.026472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.032965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2692.635308] Lustre: server umount lustre-MDT0000 complete [ 2695.141756] LustreError: 92328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.156431] LustreError: 92328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2696.683222] LustreError: 89781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339475 with bad export cookie 7021249086383899514 [ 2696.686325] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2696.702777] LustreError: 89781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2697.260597] Lustre: server umount lustre-MDT0001 complete [ 2711.728981] Lustre: server umount lustre-OST0000 complete [ 2725.407861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.066681] LustreError: 93765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.350163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.490404] LustreError: 93766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.597242] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2736.613746] LustreError: 93765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.350066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.802942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2748.826992] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2757.693942] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2764.214469] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2781.435767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2781.742438] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2781.753703] LustreError: 93754:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2781.801119] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2781.824663] LustreError: 95625:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -98 [ 2781.836965] LustreError: 95625:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2781.846098] LustreError: 95625:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2781.953691] Lustre: server umount lustre-OST0000 complete [ 2781.955915] LustreError: 95625:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2791.645939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2791.823924] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2791.842797] LustreError: 93755:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2791.881536] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2791.890957] LustreError: 96091:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -98 [ 2791.898826] LustreError: 96091:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2791.904602] LustreError: 96091:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2791.962932] Lustre: server umount lustre-OST0000 complete [ 2791.967429] LustreError: 96091:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2802.807720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2803.078165] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2803.087296] Lustre: Found index 0 for lustre-OST0000, updating log [ 2803.096603] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2803.235221] Lustre: lustre-OST0000: new disk, initializing [ 2803.243645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2803.255731] Lustre: 96626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2803.545206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2810.784376] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2815.018151] Lustre: Failing over lustre-OST0000 [ 2815.265928] Lustre: server umount lustre-OST0000 complete [ 2820.577250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2820.583767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2820.597723] Lustre: Skipped 1 previous similar message [ 2820.606871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2820.618544] Lustre: Skipped 2 previous similar messages [ 2824.679543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.683451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.749974] Lustre: server umount lustre-MDT0000 complete [ 2829.796663] LustreError: 93765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.828276] LustreError: 93765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2830.044440] LustreError: 93747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339609 with bad export cookie 7021249086383902258 [ 2830.065684] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2830.070639] LustreError: 93747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2840.863247] Lustre: server umount lustre-MDT0001 complete [ 2852.261699] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 14:47:09 (1780339629) [ 2870.413792] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2881.787526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.312586] LustreError: 98941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.453973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2887.668875] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2898.006181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2903.624889] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2906.975752] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2914.856970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2915.734479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.743647] Lustre: Skipped 1 previous similar message [ 2915.752902] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2922.630482] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2924.016413] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2924.074185] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2924.076400] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2932.287215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2932.384553] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2932.507717] Lustre: lustre-OST0001: new disk, initializing [ 2932.514241] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2932.520229] Lustre: 101212:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2939.002281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2939.019278] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2939.071916] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2939.137673] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2947.494480] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2956.933771] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2962.754321] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.59@tcp (stopping) [ 2962.772075] Lustre: Skipped 1 previous similar message [ 2964.967730] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.986591] Lustre: Skipped 1 previous similar message [ 2967.851985] LustreError: 98938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.887132] LustreError: 98938:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2967.991907] Lustre: server umount lustre-MDT0001 complete [ 2982.145405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2982.326387] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2982.335334] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2982.440341] Lustre: 98928:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2983.402989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2983.418121] Lustre: Skipped 1 previous similar message [ 2983.912843] LustreError: 102734:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2983.917752] LustreError: 102734:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2983.929434] LustreError: 98933:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2984.066293] LustreError: 102740:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.159@tcp: cfg command failed: rc = -17 [ 2984.073872] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2984.073872] [ 2988.556363] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2990.354353] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3003.037700] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 14:49:40 (1780339780) [ 3023.715803] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3042.175673] Lustre: 104778:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3064.741959] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3068.733784] Lustre: 105917:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3167.218347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3167.230557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.260542] Lustre: Skipped 1 previous similar message [ 3167.277277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3167.287907] Lustre: Skipped 3 previous similar messages [ 3170.739142] Lustre: server umount lustre-MDT0000 complete [ 3172.834671] LustreError: 98937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3172.875452] LustreError: 98937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 3175.979576] LustreError: 98922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339954 with bad export cookie 7021249086383903259 [ 3175.980924] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3176.002360] LustreError: 98922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3176.500763] Lustre: server umount lustre-MDT0001 complete [ 3191.795960] Lustre: server umount lustre-OST0000 complete [ 3193.632220] Lustre: 89114:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780339956/real 1780339956] req@ffff9a210b18ca80 x1866821091577088/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780339972 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3193.660953] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3193.669570] Lustre: Skipped 3 previous similar messages [ 3195.690240] Lustre: server umount lustre-OST0001 complete [ 3202.159659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3211.973395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3222.506446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3234.800249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3259.554677] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3271.562794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3271.584986] Lustre: lustre-MDT0000: reset Object Index mappings [ 3272.263873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.710403] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3287.942336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3287.975439] Lustre: lustre-MDT0001: reset Object Index mappings [ 3294.589557] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3298.267597] Lustre: 110972:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3306.095532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3313.129016] LustreError: 111326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3313.147713] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3313.158445] LustreError: 111326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3314.022805] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3317.264532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3324.492701] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3330.561578] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3330.563741] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3332.007513] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3339.776527] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3343.808868] Lustre: 112815:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3399.499436] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:56:17 (1780340177) [ 3426.790508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3426.799689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.818712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3426.842929] Lustre: Skipped 3 previous similar messages [ 3432.824615] Lustre: server umount lustre-MDT0000 complete [ 3438.706939] LustreError: 109835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340217 with bad export cookie 7021249086383950131 [ 3438.725664] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3438.734263] LustreError: 109835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3445.331534] Lustre: server umount lustre-MDT0001 complete [ 3464.160508] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3464.479540] Lustre: server umount lustre-OST0000 complete [ 3475.797682] Lustre: server umount lustre-OST0001 complete [ 3483.875223] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 3496.380803] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3545.329040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3545.588678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3545.645176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3545.793409] Lustre: lustre-MDT0000: new disk, initializing [ 3546.110111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.120650] Lustre: Skipped 3 previous similar messages [ 3546.175754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3556.049649] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3568.067493] Lustre: server umount lustre-MDT0000 complete [ 3578.485416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3578.963451] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3578.976807] Lustre: 117038:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3579.018907] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3584.626623] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3590.664581] Lustre: server umount lustre-MDT0000 complete [ 3600.927348] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 14:59:38 (1780340378) [ 3603.284625] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3605.884203] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 14:59:43 (1780340383) [ 3627.388350] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3640.282593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3646.151884] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3655.759374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.951631] Lustre: 119059:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3656.015774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3656.020443] Lustre: Skipped 1 previous similar message [ 3656.100871] Lustre: lustre-MDT0001: new disk, initializing [ 3656.329164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3656.342995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3661.508737] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3667.178941] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3675.084487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3675.507249] Lustre: lustre-OST0000: new disk, initializing [ 3675.516932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3675.526470] Lustre: 120602:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3677.074158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3677.099197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3677.225883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3684.256878] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3696.695879] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3696.904471] Lustre: lustre-OST0001: new disk, initializing [ 3696.910445] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3696.920133] Lustre: 121467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3698.149829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3698.165956] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3698.353088] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3704.766597] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3716.949060] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3721.387778] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4151.687726] Lustre: 127877:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4151.706657] Lustre: 127877:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4151.714560] Lustre: 127877:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4151.725701] Lustre: 127877:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4151.735420] Lustre: 127877:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4151.745652] Lustre: 127877:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4475.541938] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:14:12 (1780341252) [ 4494.818428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.835449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4494.858134] Lustre: Skipped 8 previous similar messages [ 4494.869376] Lustre: Skipped 8 previous similar messages [ 4498.167307] Lustre: server umount lustre-MDT0000 complete [ 4499.945647] LustreError: 122237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.981602] LustreError: 122237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 4503.070268] LustreError: 119791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341282 with bad export cookie 7021249086384018780 [ 4503.075885] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4503.079605] LustreError: 119791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4505.061929] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4505.063925] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4505.076598] Lustre: Skipped 1 previous similar message [ 4505.102073] Lustre: Skipped 2 previous similar messages [ 4510.004422] Lustre: server umount lustre-MDT0001 complete [ 4520.802498] Lustre: server umount lustre-OST0000 complete [ 4531.648812] Lustre: server umount lustre-OST0001 complete [ 4538.338851] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 4548.846769] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 4588.072310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.417293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4588.513422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4588.657597] Lustre: lustre-MDT0000: new disk, initializing [ 4588.760927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.767615] Lustre: Skipped 5 previous similar messages [ 4588.793331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4593.505636] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4604.791742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.924910] Lustre: 130903:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4604.973722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4604.979553] Lustre: Skipped 1 previous similar message [ 4605.057201] Lustre: lustre-MDT0001: new disk, initializing [ 4605.170888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4605.185745] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4609.634266] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4616.909722] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4621.159403] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4627.757127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4627.991662] Lustre: lustre-OST0000: new disk, initializing [ 4627.994793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4627.999969] Lustre: 132649:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4629.702130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4629.726887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4629.850554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4634.117661] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4642.154630] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4652.570706] Lustre: lustre-MDD0000: changelog on [ 4655.928508] Lustre: lustre-MDD0001: changelog on [ 6219.465708] Lustre: lustre-MDD0001: changelog off [ 6221.614246] Lustre: 130911:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6222.424916] Lustre: lustre-MDD0000: changelog off [ 6232.549780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6232.552527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6232.558135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6232.558144] Lustre: Skipped 2 previous similar messages [ 6232.610588] Lustre: Skipped 1 previous similar message [ 6237.671901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6237.691739] Lustre: Skipped 2 previous similar messages [ 6242.272284] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6242.585110] Lustre: server umount lustre-OST0000 complete [ 6247.906168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6247.925148] Lustre: Skipped 1 previous similar message [ 6247.925801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6247.939517] Lustre: Skipped 1 previous similar message [ 6252.804111] Lustre: server umount lustre-MDT0000 complete [ 6253.027321] LustreError: 130916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6253.044040] LustreError: 130916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6256.317528] LustreError: 131683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343035 with bad export cookie 7021249086384843296 [ 6256.322532] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6256.326022] LustreError: 131683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6256.671873] Lustre: server umount lustre-MDT0001 complete [ 6265.829724] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6268.160925] Key type lgssc unregistered [ 6268.380656] LNet: 141588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6268.388645] LNetError: 141588:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6268.400635] LNet: Removed LNI 192.168.204.159@tcp [ 6269.023180] Key type .llcrypt unregistered [ 6269.026894] Key type ._llcrypt unregistered [ 6285.512985] Key type ._llcrypt registered [ 6285.515360] Key type .llcrypt registered [ 6285.594066] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 6300.045750] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6301.169872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6301.385286] alg: No test for adler32 (adler32-zlib) [ 6302.602282] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 6302.944064] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 6304.640469] Key type lgssc registered [ 6305.855758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6338.330250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6338.352893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6339.613421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6339.683307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6339.742264] Lustre: lustre-MDT0000: new disk, initializing [ 6339.823371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6339.838748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6343.946406] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6353.544317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6353.668591] Lustre: 144755:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6353.732872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6353.740461] Lustre: Skipped 1 previous similar message [ 6353.820667] Lustre: lustre-MDT0001: new disk, initializing [ 6353.934213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6353.979956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6353.991680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6358.018418] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6366.310319] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6370.409195] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6377.176056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6377.448436] Lustre: lustre-OST0000: new disk, initializing [ 6377.454122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6377.462987] Lustre: 146498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6377.583731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6383.721659] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6385.300308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6385.312340] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6385.473022] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6392.736707] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6397.989383] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_module obdecho/obdecho [ 6400.362069] LustreError: 147375:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6400.369948] LustreError: 147375:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6402.723496] LustreError: 147472:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6415.329627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6415.341844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6415.351335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6416.353885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6416.358784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6416.385358] Lustre: Skipped 1 previous similar message [ 6420.691046] Lustre: server umount lustre-MDT0000 complete [ 6425.737285] LustreError: 144748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343204 with bad export cookie 12824504355184850295 [ 6425.739539] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6425.744646] LustreError: 148014:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a020a927 x1866825026546688/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6425.744749] LustreError: 148014:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x20000000a:0x0:0x0]: rc = -5 [ 6425.744832] LustreError: 148014:0:(llog_cat.c:443:llog_cat_close()) lustre-MDT0000-osp-MDT0001: failure destroying log during cleanup: rc = -5 [ 6425.760127] LustreError: 144748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6426.321893] Lustre: server umount lustre-MDT0001 complete [ 6440.519873] Lustre: server umount lustre-OST0000 complete [ 6463.546376] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 6473.743893] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6511.903067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6512.166407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6512.209252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6512.309442] Lustre: lustre-MDT0000: new disk, initializing [ 6512.380361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6512.396603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6516.372900] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6526.417746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6526.508117] Lustre: 151152:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6526.557276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6526.561211] Lustre: Skipped 1 previous similar message [ 6526.630821] Lustre: lustre-MDT0001: new disk, initializing [ 6526.712857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6526.776974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6526.798601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6531.239862] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6539.094175] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6542.790551] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6549.608705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6549.845534] Lustre: lustre-OST0000: new disk, initializing [ 6549.848755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6549.854711] Lustre: 152897:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6549.927687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6556.080135] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6557.752868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6557.761383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6557.903445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6564.089375] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6568.826301] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6569.079214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6573.994160] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6574.244810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6578.659926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6578.660351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6578.660796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6578.682051] Lustre: Skipped 2 previous similar messages [ 6582.333316] Lustre: server umount lustre-OST0000 complete [ 6588.399316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6588.413184] LustreError: Skipped 1 previous similar message [ 6588.422962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6588.479639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6588.486046] Lustre: Skipped 1 previous similar message [ 6592.959237] Lustre: server umount lustre-MDT0000 complete [ 6594.019230] LustreError: 151164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6594.043201] LustreError: 151164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6597.067980] LustreError: 151939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343376 with bad export cookie 12824504355184851289 [ 6597.070981] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6597.092605] LustreError: 151939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6597.539329] Lustre: server umount lustre-MDT0001 complete [ 6601.902070] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:49:39 (1780343379) [ 6603.597841] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6605.657916] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 15:49:43 (1780343383) [ 6614.233180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6614.660773] LustreError: 155159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6614.749540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6619.262080] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6620.139261] LustreError: 155160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6624.239571] LustreError: 155159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6628.015518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6633.081346] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6638.681219] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6642.460333] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6649.055254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6649.392270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6649.401138] Lustre: Skipped 1 previous similar message [ 6655.423734] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6661.143360] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6668.157809] Lustre: server umount lustre-MDT0000 complete [ 6669.281815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6669.293246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.308021] Lustre: Skipped 1 previous similar message [ 6669.313313] LustreError: 155872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6671.755647] LustreError: 155140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343450 with bad export cookie 12824504355184852220 [ 6671.764604] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6671.770091] LustreError: 155140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6672.172042] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.59@tcp (stopping) [ 6672.180465] Lustre: Skipped 1 previous similar message [ 6681.059972] LustreError: 156521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6681.064796] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6681.078270] LustreError: 156521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6681.092188] Lustre: Skipped 3 previous similar messages [ 6686.176726] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6686.480874] Lustre: server umount lustre-MDT0001 complete [ 6692.125628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6703.714686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6703.900031] LustreError: 158675:0:(llog.c:1646:llog_backup()) MGC192.168.204.159@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6703.906442] Lustre: 158675:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.204.159@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6712.078650] LustreError: 158680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6712.098297] LustreError: 158680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6744.240208] LustreError: 156767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6744.263914] LustreError: 156767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 6764.512301] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6764.518361] Lustre: 158675:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.159@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6805.495309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6809.023631] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6809.353228] LustreError: 158680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6809.370973] LustreError: 158680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 39 previous similar messages [ 6810.598934] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6816.213028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6816.737410] LustreError: 142977:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001e096d84 x1866825026635264/t0(0) o250->MGC192.168.204.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6820.316992] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6822.394705] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6825.625563] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6827.159395] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6830.435181] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6831.758427] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6847.970501] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6847.984520] Lustre: Skipped 4 previous similar messages [ 6847.992331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6848.034504] Lustre: server umount lustre-MDT0000 complete [ 6850.874305] LustreError: 155140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343629 with bad export cookie 12824504355184858674 [ 6850.891387] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6850.891800] LustreError: 155140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6851.222277] Lustre: server umount lustre-MDT0001 complete [ 6864.547463] Lustre: server umount lustre-OST0000 complete [ 6881.106575] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 6889.718633] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6918.337886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.516426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6918.545445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6918.612131] Lustre: lustre-MDT0000: new disk, initializing [ 6918.678859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6921.831864] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6930.220656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.344174] Lustre: 163514:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6930.572841] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6934.095486] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6940.250321] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6943.063996] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6948.516610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6948.729931] Lustre: 165259:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6948.808462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6948.819737] Lustre: Skipped 3 previous similar messages [ 6953.886256] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6958.668628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6958.681920] Lustre: Skipped 1 previous similar message [ 6958.689645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6958.861403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6960.594481] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6964.447728] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6964.674556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6969.001071] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6969.180393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6974.437592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6974.449243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.453905] LustreError: Skipped 1 previous similar message [ 6974.466081] Lustre: Skipped 1 previous similar message [ 6974.467465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6974.482950] Lustre: Skipped 3 previous similar messages [ 6977.338633] Lustre: server umount lustre-OST0000 complete [ 6987.150789] Lustre: server umount lustre-MDT0000 complete [ 6989.794735] LustreError: 163527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6989.808063] LustreError: 163527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 6990.489450] LustreError: 163507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343769 with bad export cookie 12824504355184875502 [ 6990.493819] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6990.500253] LustreError: 163507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6990.855415] Lustre: server umount lustre-MDT0001 complete [ 6998.091215] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:56:16 (1780343776) [ 7003.068633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7006.604708] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7013.574833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7017.672630] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7022.665279] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7025.742421] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7031.644561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7037.042974] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7041.516091] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7059.946793] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:57:18 (1780343838) [ 7064.034681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7064.050580] Lustre: Skipped 3 previous similar messages [ 7064.058195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7064.058797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7064.064836] LustreError: Skipped 1 previous similar message [ 7064.081748] Lustre: Skipped 4 previous similar messages [ 7069.472687] Lustre: server umount lustre-OST0000 complete [ 7078.648321] Lustre: server umount lustre-MDT0000 complete [ 7081.656493] LustreError: 167501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343860 with bad export cookie 12824504355184876419 [ 7081.659503] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7081.666108] LustreError: 167501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7081.875662] Lustre: server umount lustre-MDT0001 complete [ 7089.828846] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7091.965973] Key type lgssc unregistered [ 7092.197945] LNet: 170919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7092.210368] LNetError: 170919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7092.223631] LNet: Removed LNI 192.168.204.159@tcp [ 7092.808784] Key type .llcrypt unregistered [ 7092.814088] Key type ._llcrypt unregistered [ 7105.921785] Key type ._llcrypt registered [ 7105.924815] Key type .llcrypt registered [ 7105.997178] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7106.614889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7106.674055] alg: No test for adler32 (adler32-zlib) [ 7107.658102] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 7107.846753] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 7109.480194] Key type lgssc registered [ 7110.382458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7116.088712] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7116.106507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7131.618259] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7136.736363] LustreError: 172123:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.159@tcp: failed processing log, type 4: rc = -110 [ 7141.857777] LustreError: 172123:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7141.864883] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7141.868743] LustreError: MGC192.168.204.159@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7141.875651] LustreError: 172123:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7141.882718] LustreError: 172123:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -22 [ 7141.889783] LustreError: 172123:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7141.933995] Lustre: server umount lustre-OST0000 complete [ 7141.936465] LustreError: 172123:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7148.537455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7149.890052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7152.503523] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7159.266191] LustreError: 172622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7159.280387] LustreError: 172622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7159.282750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7159.608429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7163.067639] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7167.713468] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7170.757868] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7176.316297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7176.457299] LustreError: 174211:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7176.468138] Lustre: 174211:0:(mgc_request_server.c:748:mgc_llog_local_copy()) MGC192.168.204.159@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7176.625367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7181.533394] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7185.725774] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7187.402283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7187.405789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7187.415580] Lustre: Skipped 2 previous similar messages [ 7187.421235] Lustre: Skipped 1 previous similar message [ 7192.547408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7192.551724] Lustre: Skipped 3 previous similar messages [ 7193.635410] Lustre: server umount lustre-MDT0000 complete [ 7196.267815] LustreError: 174238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343975 with bad export cookie 9971829464675779312 [ 7196.271773] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7196.277819] LustreError: 174238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7196.495613] Lustre: server umount lustre-MDT0001 complete [ 7209.699923] Lustre: server umount lustre-OST0000 complete [ 7217.096920] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 7225.700551] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7254.825498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7255.018946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7255.065600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7255.141143] Lustre: lustre-MDT0000: new disk, initializing [ 7255.200611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7255.221757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7258.539993] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7267.514968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.616748] Lustre: 177588:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7267.678507] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7267.686759] Lustre: Skipped 1 previous similar message [ 7267.781219] Lustre: lustre-MDT0001: new disk, initializing [ 7267.891828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7267.927160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7267.936856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7271.592863] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7278.101092] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7281.584684] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7287.973669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7288.231502] Lustre: lustre-OST0000: new disk, initializing [ 7288.234746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7288.240857] Lustre: 179344:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7288.339082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7293.767627] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7298.095243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7298.113494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7298.209922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7300.472937] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7303.946287] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7304.145498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7307.823060] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7308.008839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7313.894099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7313.895403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7313.899262] LustreError: Skipped 1 previous similar message [ 7313.900554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7313.910221] Lustre: Skipped 1 previous similar message [ 7315.603056] Lustre: server umount lustre-OST0000 complete [ 7319.014237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7319.015494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7319.027064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7319.043444] Lustre: Skipped 2 previous similar messages [ 7324.135281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7324.140713] Lustre: Skipped 1 previous similar message [ 7324.819795] Lustre: server umount lustre-MDT0000 complete [ 7327.977716] LustreError: 177581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344106 with bad export cookie 9971829464675780026 [ 7327.985101] LustreError: 177581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7327.993612] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7328.002239] LustreError: 171461:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting [ 7328.323108] Lustre: server umount lustre-MDT0001 complete [ 7335.801709] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 16:01:53 (1780344113) [ 7348.840496] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7351.333066] Key type lgssc unregistered [ 7351.587722] LNet: 182123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7351.595849] LNetError: 182123:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7351.612600] LNet: Removed LNI 192.168.204.159@tcp [ 7352.298373] Key type .llcrypt unregistered [ 7352.300894] Key type ._llcrypt unregistered [ 7365.004427] Key type ._llcrypt registered [ 7365.006566] Key type .llcrypt registered [ 7365.082514] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7365.802446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7365.841622] alg: No test for adler32 (adler32-zlib) [ 7366.805107] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 7366.960527] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 7368.624386] Key type lgssc registered [ 7369.419208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7375.032570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7375.061624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7376.447723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7380.008707] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7386.595104] LustreError: 183305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7386.608725] LustreError: 183305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7387.652279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7387.954728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7391.614649] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7395.794425] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7398.744083] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7405.059222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7405.218952] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7405.221508] LustreError: 184942:0:(llog.c:1687:llog_backup()) MGC192.168.204.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7405.228404] Lustre: 184942:0:(mgc_request_server.c:745:mgc_llog_local_copy()) MGC192.168.204.159@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7405.341224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7409.556952] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7413.580240] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7417.827777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7417.838536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7417.840411] Lustre: Skipped 2 previous similar messages [ 7417.852773] Lustre: Skipped 2 previous similar messages [ 7421.390746] Lustre: server umount lustre-MDT0000 complete [ 7422.949107] LustreError: 183304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7422.967384] LustreError: 183304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7424.059249] LustreError: 183285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344203 with bad export cookie 10423934324752134597 [ 7424.072423] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7424.076069] LustreError: 183285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7424.325217] Lustre: server umount lustre-MDT0001 complete [ 7438.049381] Lustre: server umount lustre-OST0000 complete [ 7445.647428] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 7454.160470] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7480.468137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7480.617876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7480.643557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7480.694763] Lustre: lustre-MDT0000: new disk, initializing [ 7480.750912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7480.770517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7483.834979] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7491.898526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7491.974903] Lustre: 188314:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7492.011237] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7492.013931] Lustre: Skipped 1 previous similar message [ 7492.016725] LustreError: 189084:0:(llog.c:1687:llog_backup()) MGC192.168.204.159@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7492.022346] LustreError: 189084:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7492.030138] Lustre: 189084:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.204.159@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7492.044055] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7492.047566] Lustre: Skipped 1 previous similar message [ 7492.096684] Lustre: lustre-MDT0001: new disk, initializing [ 7492.154388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7492.181978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7492.187696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7495.155666] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7500.830773] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7503.154287] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7508.144522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7508.335920] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7508.337897] Lustre: Skipped 1 previous similar message [ 7508.343621] LustreError: 190037:0:(llog.c:1687:llog_backup()) MGC192.168.204.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7508.349479] LustreError: 190037:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7508.353244] Lustre: 190037:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.204.159@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7508.359081] Lustre: 190037:0:(mgc_request_server.c:768:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7508.373267] Lustre: lustre-OST0000: new disk, initializing [ 7508.376405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7508.380948] Lustre: 190061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7508.455528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7512.471632] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7514.150340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7514.157662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7514.219615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7517.644509] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7520.758203] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7520.882932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7523.496394] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7523.659087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7529.443721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7529.444817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7529.452837] LustreError: Skipped 1 previous similar message [ 7529.456028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7529.462555] Lustre: Skipped 1 previous similar message [ 7529.483945] Lustre: Skipped 1 previous similar message [ 7534.570157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7534.575787] Lustre: Skipped 1 previous similar message [ 7539.168754] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7539.275488] Lustre: server umount lustre-OST0000 complete [ 7543.265761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7543.272693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.279658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7548.049145] Lustre: server umount lustre-MDT0000 complete [ 7549.923320] LustreError: 188326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7549.938221] LustreError: 188326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7550.285504] LustreError: 188306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344329 with bad export cookie 10423934324752135311 [ 7550.287702] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7550.296666] LustreError: 188306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7550.548567] Lustre: server umount lustre-MDT0001 complete [ 7555.811430] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 16:05:34 (1780344334) [ 7565.748194] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7572.805356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7573.103472] LustreError: 193019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7573.152640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7575.914311] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7578.594058] LustreError: 193020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7581.380767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7584.083665] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7585.893246] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7590.074336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7594.075445] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7599.908705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7600.050463] Lustre: lustre-OST0001: new disk, initializing [ 7600.053982] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7600.059243] Lustre: 195279:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7602.372179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7602.377255] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7602.443128] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7604.364777] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7612.439739] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7617.906416] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7619.190637] Lustre: DEBUG MARKER: ADD OST3 [ 7622.148453] loop: module loaded [ 7622.197413] loop0: detected capacity change from 0 to 204800000 [ 7622.318959] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7622.337174] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7622.351919] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7626.652263] loop0: detected capacity change from 0 to 204800000 [ 7628.509888] Lustre: DEBUG MARKER: STOP OST3 [ 7630.542163] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7630.653394] Lustre: lustre-OST0003: new disk, initializing [ 7630.655778] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7630.659931] Lustre: 197190:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7630.748144] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7630.759427] Lustre: Skipped 3 previous similar messages [ 7631.831826] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7631.916445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7631.930924] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7631.965977] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7632.016127] Lustre: Failing over lustre-OST0003 [ 7632.099018] Lustre: server umount lustre-OST0003 complete [ 7632.510246] LustreError: 194474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7633.889241] LustreError: 197354:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7634.805646] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7636.972863] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7636.983555] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7636.995499] Lustre: Skipped 2 previous similar messages [ 7639.993671] Lustre: DEBUG MARKER: START OST3 again [ 7644.270477] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7644.440572] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7645.477269] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7645.495952] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7645.496251] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7647.885383] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7664.322081] Lustre: Failing over lustre-OST0003 [ 7664.398642] Lustre: server umount lustre-OST0003 complete [ 7666.147590] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7666.153907] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7666.156849] LustreError: 195155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7666.166403] Lustre: Skipped 1 previous similar message [ 7666.178309] LustreError: 195155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 7683.560958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7683.564164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7683.568095] Lustre: Skipped 3 previous similar messages [ 7683.572070] Lustre: Skipped 4 previous similar messages [ 7685.374889] Lustre: server umount lustre-MDT0000 complete [ 7687.767060] LustreError: 193001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344466 with bad export cookie 10423934324752136228 [ 7687.771472] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7687.774810] LustreError: 193001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7692.710490] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7692.716741] Lustre: Skipped 3 previous similar messages [ 7694.002540] Lustre: server umount lustre-MDT0001 complete [ 7696.215596] Lustre: server umount lustre-OST0000 complete [ 7698.487395] Lustre: server umount lustre-OST0001 complete [ 7702.300936] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 7706.857695] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7727.419576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7727.546851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7727.567965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7727.613451] Lustre: lustre-MDT0000: new disk, initializing [ 7727.665805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7727.671587] Lustre: Skipped 1 previous similar message [ 7727.682939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7729.715225] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7735.845406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7735.918134] Lustre: 202492:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7736.118423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7738.462076] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7742.961553] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7745.120149] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7749.676634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7749.865324] Lustre: lustre-OST0000: new disk, initializing [ 7749.867624] Lustre: Skipped 1 previous similar message [ 7749.870390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7749.875571] Lustre: Skipped 2 previous similar messages [ 7749.881905] Lustre: 204235:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7751.640274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7751.645740] Lustre: Skipped 1 previous similar message [ 7751.650384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7751.700608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7753.237299] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7757.783897] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7760.726665] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7760.873569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7763.182625] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7763.324305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7767.009802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7767.010527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7767.016785] LustreError: Skipped 2 previous similar messages [ 7767.026096] Lustre: Skipped 2 previous similar messages [ 7767.028163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7767.038339] Lustre: Skipped 1 previous similar message [ 7770.485079] Lustre: server umount lustre-OST0000 complete [ 7777.251035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7778.889546] Lustre: server umount lustre-MDT0000 complete [ 7781.050950] LustreError: 202484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344560 with bad export cookie 10423934324752138524 [ 7781.056995] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7781.061939] LustreError: 202484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7781.277818] Lustre: server umount lustre-MDT0001 complete [ 7783.461355] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 16:09:22 (1780344562) [ 7793.074952] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 7799.240582] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7821.134100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7821.291272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7821.315929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7821.378852] Lustre: lustre-MDT0000: new disk, initializing [ 7821.434396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7823.722718] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7830.302062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7830.394826] Lustre: 208657:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7830.608204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7833.030897] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7837.600959] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7839.787969] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7843.780557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7843.968733] Lustre: 210401:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7845.496958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7847.470767] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7852.186660] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7854.693474] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7854.818247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7857.017158] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7857.156785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7860.708927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7860.711406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7860.720323] Lustre: Skipped 3 previous similar messages [ 7860.721421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7860.721427] Lustre: Skipped 2 previous similar messages [ 7864.418356] Lustre: server umount lustre-OST0000 complete [ 7881.184217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7881.305073] Lustre: server umount lustre-MDT0000 complete [ 7883.237462] LustreError: 210400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344662 with bad export cookie 10423934324752139448 [ 7883.240888] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7883.244956] LustreError: 210400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7883.413190] Lustre: server umount lustre-MDT0001 complete [ 7888.052511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7888.290580] LustreError: 212469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7888.303373] LustreError: 212469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 7888.340922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7888.346191] Lustre: Skipped 5 previous similar messages [ 7890.431477] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7895.225150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7897.733424] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7900.891115] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7903.474064] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7907.265717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7910.821292] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7914.071488] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7929.315054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7929.321867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7929.336093] Lustre: Skipped 3 previous similar messages [ 7929.341709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7929.348505] Lustre: Skipped 7 previous similar messages [ 7933.352214] Lustre: server umount lustre-OST0000 complete [ 7941.532835] Lustre: server umount lustre-MDT0000 complete [ 7943.567477] LustreError: 213188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344722 with bad export cookie 10423934324752140365 [ 7943.570967] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7943.580661] LustreError: 213188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7943.764293] Lustre: server umount lustre-MDT0001 complete [ 8049.463312] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 8051.088512] Key type lgssc unregistered [ 8051.273392] LNet: 215798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8051.278639] LNetError: 215798:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8051.294101] LNet: Removed LNI 192.168.204.159@tcp [ 8051.726156] Key type .llcrypt unregistered [ 8051.728205] Key type ._llcrypt unregistered [ 8064.832660] Key type ._llcrypt registered [ 8064.834702] Key type .llcrypt registered [ 8064.888602] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 8073.104835] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 8073.486079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8073.500410] alg: No test for adler32 (adler32-zlib) [ 8074.407340] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 8074.533790] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 8076.144243] Key type lgssc registered [ 8076.744293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8095.168279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8095.183982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8096.333817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8096.354930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8096.397922] Lustre: lustre-MDT0000: new disk, initializing [ 8096.444492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8096.455143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8098.634913] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8104.596874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8104.652238] Lustre: 219091:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8104.684415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8104.686919] Lustre: Skipped 1 previous similar message [ 8104.765134] Lustre: lustre-MDT0001: new disk, initializing [ 8104.819405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8104.838954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8104.842752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8106.832894] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8111.350494] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8113.014075] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8116.137556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8116.263453] Lustre: lustre-OST0000: new disk, initializing [ 8116.266674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8116.270832] Lustre: 220838:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8116.311175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8117.772164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8117.775319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8117.830890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8118.962910] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8123.194864] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8125.345753] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8125.450596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8127.457263] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8127.555180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8133.090608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8133.092555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8133.095554] LustreError: Skipped 1 previous similar message [ 8133.105776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8133.115011] Lustre: Skipped 1 previous similar message [ 8134.564438] Lustre: server umount lustre-OST0000 complete [ 8138.211878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8138.218189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8138.218900] Lustre: Skipped 2 previous similar messages [ 8138.225934] Lustre: Skipped 1 previous similar message [ 8142.451987] Lustre: server umount lustre-MDT0000 complete [ 8144.180166] LustreError: 219085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344923 with bad export cookie 6543022406577913427 [ 8144.182657] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8144.185427] LustreError: 219085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8144.359861] Lustre: server umount lustre-MDT0001 complete [ 8153.846747] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 8159.550768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8159.821815] LustreError: 223602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8159.835640] LustreError: 223602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8159.856884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8162.076121] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8165.346234] LustreError: 223603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8166.850331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8167.069193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8169.857646] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8171.665515] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8175.339127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8175.547075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8178.601377] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8183.203830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8183.300779] Lustre: lustre-OST0001: new disk, initializing [ 8183.303713] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8183.307632] Lustre: 225860:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8186.538115] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8188.976750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8188.983049] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8189.047542] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8193.270587] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8196.645508] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8202.614565] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 16:16:21 (1780344981) [ 8204.263732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8204.264842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8204.276437] Lustre: Skipped 3 previous similar messages [ 8204.280436] Lustre: Skipped 2 previous similar messages [ 8205.766308] Lustre: server umount lustre-MDT0000 complete [ 8207.615967] LustreError: 223584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344986 with bad export cookie 6543022406577914351 [ 8207.619282] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8207.626655] LustreError: 223584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8207.802230] Lustre: server umount lustre-MDT0001 complete [ 8220.111084] Lustre: server umount lustre-OST0000 complete [ 8231.217118] Lustre: server umount lustre-OST0001 complete [ 8234.071923] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 8238.218000] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 8254.456934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8254.589829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8254.607994] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8254.656729] Lustre: lustre-MDT0000: new disk, initializing [ 8254.710273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8254.712667] Lustre: Skipped 1 previous similar message [ 8254.722091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8256.548266] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8262.015553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8262.067034] Lustre: 229751:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8262.205705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8263.980448] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8267.809692] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8269.433174] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8272.285202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8272.404115] Lustre: lustre-OST0000: new disk, initializing [ 8272.406146] Lustre: Skipped 1 previous similar message [ 8272.408694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8272.411370] Lustre: Skipped 2 previous similar messages [ 8272.414262] Lustre: 231497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8274.383477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8274.387373] Lustre: Skipped 1 previous similar message [ 8274.389836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8274.455684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8275.105589] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8278.936489] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8280.863922] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8280.966811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8282.811116] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8282.913258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8284.645916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8284.646366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8284.652415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8284.655812] LustreError: Skipped 1 previous similar message [ 8284.658510] Lustre: Skipped 2 previous similar messages [ 8289.764360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8289.770238] Lustre: Skipped 1 previous similar message [ 8289.949428] Lustre: server umount lustre-OST0000 complete [ 8292.833096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8292.836215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8292.842213] Lustre: Skipped 1 previous similar message [ 8297.846290] Lustre: server umount lustre-MDT0000 complete [ 8299.548642] LustreError: 229741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345078 with bad export cookie 6543022406577916472 [ 8299.551688] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8299.554198] LustreError: 229741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8299.701895] Lustre: server umount lustre-MDT0001 complete [ 8303.442628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8303.646085] LustreError: 233565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8303.685685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8303.688873] Lustre: Skipped 2 previous similar messages [ 8305.376963] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8308.706588] LustreError: 233565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8308.875330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8310.638810] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8312.875794] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8314.434318] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8317.323659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8319.629798] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8321.932526] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8472.035681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8472.036376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8472.037105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8472.037109] Lustre: Skipped 3 previous similar messages [ 8472.041131] Lustre: Skipped 2 previous similar messages [ 8472.044823] LustreError: Skipped 1 previous similar message [ 8482.272184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8482.346791] Lustre: server umount lustre-OST0000 complete [ 8490.279417] Lustre: server umount lustre-MDT0000 complete [ 8491.944636] LustreError: 233545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345270 with bad export cookie 6543022406577917396 [ 8491.948783] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8491.950981] LustreError: 233545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8492.090175] Lustre: server umount lustre-MDT0001 complete [ 8593.428347] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 8594.757704] Key type lgssc unregistered [ 8594.894786] LNet: 236819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8594.899838] LNetError: 236819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8594.911781] LNet: Removed LNI 192.168.204.159@tcp [ 8595.324199] Key type .llcrypt unregistered [ 8595.326464] Key type ._llcrypt unregistered [ 8606.713078] Key type ._llcrypt registered [ 8606.714156] Key type .llcrypt registered [ 8606.765346] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 8614.763364] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 8615.188557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8615.234471] alg: No test for adler32 (adler32-zlib) [ 8616.152824] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 8616.272466] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 8617.888195] Key type lgssc registered [ 8618.413601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8639.433043] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 8643.857064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8643.867977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8644.994240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8645.011712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8645.063473] Lustre: lustre-MDT0000: new disk, initializing [ 8645.108566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8645.119932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8646.815670] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8652.414394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8652.469900] Lustre: 240810:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8652.523513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8652.526252] Lustre: Skipped 1 previous similar message [ 8652.566691] Lustre: lustre-MDT0001: new disk, initializing [ 8652.618119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8652.639372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8652.647272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8654.449169] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8657.104940] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8659.798576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8659.917485] Lustre: lustre-OST0000: new disk, initializing [ 8659.919858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8659.923710] Lustre: 242410:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8659.964302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8662.284701] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8665.094871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8665.101312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8665.145539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8667.469488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8667.562806] Lustre: lustre-OST0001: new disk, initializing [ 8667.565768] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8667.569559] Lustre: 243263:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8667.619802] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8670.396959] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8674.324791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8674.329450] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8674.371095] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8676.782526] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8680.024648] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8688.609770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8688.615613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8688.623514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8689.634339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8689.637206] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8689.645571] Lustre: Skipped 2 previous similar messages [ 8690.811652] Lustre: server umount lustre-MDT0000 complete [ 8692.435987] LustreError: 241587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345471 with bad export cookie 15308957485250291338 [ 8692.441214] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8692.443657] LustreError: 241587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8692.604145] Lustre: server umount lustre-MDT0001 complete [ 8704.975380] Lustre: server umount lustre-OST0000 complete [ 8716.774038] Lustre: server umount lustre-OST0001 complete [ 8722.200296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8726.249599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8729.671981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8733.703973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8740.974064] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 8745.185825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8745.194751] Lustre: lustre-MDT0000: reset Object Index mappings [ 8745.407453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8746.958820] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8750.561557] LustreError: 248227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8750.571498] LustreError: 248227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8750.750201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8750.759532] Lustre: lustre-MDT0001: reset Object Index mappings [ 8752.506613] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8753.666940] Lustre: 249337:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8756.357544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8756.503224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8756.508018] Lustre: Skipped 1 previous similar message [ 8758.756561] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8760.613282] LustreError: 250261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8762.163933] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8764.502489] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8771.566185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8771.566793] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8778.897182] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8780.269494] Lustre: 251190:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8781.795699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8781.797027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8781.797380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8781.797387] Lustre: Skipped 3 previous similar messages [ 8781.800040] Lustre: Skipped 2 previous similar messages [ 8786.915280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8786.917758] Lustre: Skipped 4 previous similar messages [ 8787.700652] Lustre: server umount lustre-MDT0000 complete [ 8789.091097] LustreError: 249339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345568 with bad export cookie 15308957485250298170 [ 8789.093949] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8789.098892] LustreError: 249339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8789.220074] Lustre: server umount lustre-MDT0001 complete [ 8800.196862] Lustre: server umount lustre-OST0000 complete [ 8812.063402] Lustre: server umount lustre-OST0001 complete [ 8819.232242] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 16:26:38 (1780345598) [ 8825.527937] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 8829.679627] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 8844.539855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8844.642752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8844.659671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8844.700561] Lustre: lustre-MDT0000: new disk, initializing [ 8844.735168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8844.737977] Lustre: Skipped 1 previous similar message [ 8844.744892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8846.239618] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8851.072455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8851.123637] Lustre: 255405:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8851.254559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8852.915957] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8856.602402] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8858.093162] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8860.969508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8861.098744] Lustre: lustre-OST0000: new disk, initializing [ 8861.101085] Lustre: Skipped 1 previous similar message [ 8861.104093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8861.106683] Lustre: Skipped 2 previous similar messages [ 8861.109044] Lustre: 257149:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8862.656631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8862.660816] Lustre: Skipped 1 previous similar message [ 8862.664271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8862.713247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8863.607315] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8867.219540] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8868.897910] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8868.993369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8870.582960] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8870.675304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8872.934937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8872.934945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8872.935915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8872.940164] Lustre: Skipped 2 previous similar messages [ 8877.610123] Lustre: server umount lustre-OST0000 complete [ 8882.145869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8882.149621] LustreError: Skipped 1 previous similar message [ 8882.151968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8882.158841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8882.161489] Lustre: Skipped 1 previous similar message [ 8885.283180] Lustre: server umount lustre-MDT0000 complete [ 8886.846868] LustreError: 255398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345665 with bad export cookie 15308957485250300242 [ 8886.849534] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8886.854055] LustreError: 255398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8886.999988] Lustre: server umount lustre-MDT0001 complete [ 8894.863440] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 8898.948854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8899.116099] LustreError: 259914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8899.136398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8899.139394] Lustre: Skipped 2 previous similar messages [ 8900.646897] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8903.930534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8905.730852] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8906.912343] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8909.559239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8912.031610] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8916.549516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8916.674946] Lustre: lustre-OST0001: new disk, initializing [ 8916.678514] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8916.683708] Lustre: 262171:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8918.058349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8918.067293] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8918.217656] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8923.036506] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8932.593773] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8936.121608] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8942.010213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8942.016238] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8953.319654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8953.327360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8953.338694] Lustre: Skipped 3 previous similar messages [ 8953.347115] Lustre: Skipped 3 previous similar messages [ 8957.858651] Lustre: server umount lustre-MDT0000 complete [ 8958.435634] LustreError: 259909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8958.460533] LustreError: 259909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8963.121396] LustreError: 259895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345742 with bad export cookie 15308957485250301166 [ 8963.132319] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8963.147498] LustreError: 259895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8963.637832] Lustre: server umount lustre-MDT0001 complete [ 8979.936313] Lustre: 238337:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780345742/real 1780345742] req@ffff9a210bd4fb80 x1866827453111168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780345758 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8979.992397] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8980.017474] Lustre: Skipped 1 previous similar message [ 8980.168543] Lustre: server umount lustre-OST0000 complete [ 8985.093671] Lustre: server umount lustre-OST0001 complete [ 9005.116482] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 9013.988251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9014.284607] LustreError: 265191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9014.345435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9014.353827] Lustre: Skipped 3 previous similar messages [ 9017.758936] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9024.486851] LustreError: 265191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9024.504522] LustreError: 265191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9024.554256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9027.944553] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9030.034226] Lustre: 266299:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9034.803667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9039.152968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9039.458298] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9044.459185] LustreError: 266653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9044.478632] LustreError: 266653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9045.336595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9049.674067] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9049.928522] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9055.600941] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9058.521176] Lustre: 268143:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9061.996454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9062.129828] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9068.988751] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 16:30:47 (1780345847) [ 9085.414963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9085.428587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9085.433036] Lustre: Skipped 2 previous similar messages [ 9085.448933] Lustre: Skipped 5 previous similar messages [ 9089.078771] Lustre: server umount lustre-MDT0000 complete [ 9090.536797] LustreError: 265187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9090.551847] LustreError: 265187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9092.214603] LustreError: 265917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345871 with bad export cookie 15308957485250307473 [ 9092.226238] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9092.227487] LustreError: 265917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9098.533885] Lustre: server umount lustre-MDT0001 complete [ 9107.360909] Lustre: server umount lustre-OST0000 complete [ 9124.320259] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9124.481629] Lustre: server umount lustre-OST0001 complete [ 9128.925590] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 9136.459781] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 9171.816392] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 9179.019936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9179.198844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9179.226912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9179.277262] Lustre: lustre-MDT0000: new disk, initializing [ 9179.350035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9179.356919] Lustre: Skipped 3 previous similar messages [ 9179.374861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9182.279886] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9189.250657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9189.343768] Lustre: 272331:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9189.518875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9192.458638] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9200.394367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9200.571345] Lustre: 273930:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9202.717431] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9204.365574] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9210.589572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9210.689448] Lustre: 274784:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9211.975229] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9214.175195] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9220.832237] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9223.230866] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9223.238287] Lustre: Skipped 1 previous similar message [ 9235.257295] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 16:33:33 (1780346013) [ 9237.477605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9237.480577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9237.485028] Lustre: Skipped 6 previous similar messages [ 9237.496881] Lustre: Skipped 7 previous similar messages [ 9243.081791] Lustre: server umount lustre-MDT0000 complete [ 9244.776286] LustreError: 272321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346023 with bad export cookie 15308957485250315341 [ 9244.779883] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9244.786171] LustreError: 272321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9244.938716] Lustre: server umount lustre-MDT0001 complete [ 9256.981329] Lustre: server umount lustre-OST0000 complete [ 9268.922237] Lustre: server umount lustre-OST0001 complete [ 9272.725161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9272.903088] LustreError: 277013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9272.912904] LustreError: 277013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9274.439076] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9277.827211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9279.610508] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9281.780661] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9283.283073] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9286.211058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9288.847075] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9291.375794] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9311.156271] loop1: detected capacity change from 0 to 204800000 [ 9311.276027] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9311.283403] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9311.296350] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9314.118184] loop1: detected capacity change from 0 to 204800000 [ 9316.800791] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9316.837189] LustreError: 277002:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs1290: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9316.844178] LustreError: 277002:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs1290 [ 9316.846802] LustreError: 277002:0:(mgs_handler.c:517:mgs_target_reg()) fs1290-MDT0001: Error checking target index: rc = -13 [ 9316.852054] LustreError: fs1290-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9316.855910] LustreError: 280464:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -13 [ 9316.859298] LustreError: 280464:0:(tgt_mount.c:2084:server_put_super()) no obd fs1290-MDT0001 [ 9316.863598] LustreError: 280464:0:(tgt_mount.c:116:server_deregister_mount()) fs1290-MDT0001 not registered [ 9316.890886] Lustre: server umount fs1290-MDT0001 complete [ 9316.893184] LustreError: 280464:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9321.245203] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9321.300706] Lustre: Setting parameter fs1290-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs1290-MDT0001 [ 9321.335552] Lustre: srv-fs1290-MDT0001: No data found on store. Initialize space. [ 9321.338798] Lustre: Skipped 4 previous similar messages [ 9321.381951] Lustre: fs1290-MDT0001: new disk, initializing [ 9321.383752] Lustre: Skipped 3 previous similar messages [ 9321.434372] Lustre: srv-fs1290-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9321.439986] Lustre: 280930:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1290-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9321.444663] LustreError: 280930:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1290-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9321.455955] LustreError: 280932:0:(fld_handler.c:275:fld_server_lookup()) srv-fs1290-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 9321.473494] LustreError: 280932:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) fs1290-MDT0001: Trash dir setup failed: rc = -5 [ 9321.955091] LustreError: 280932:0:(fld_handler.c:275:fld_server_lookup()) srv-fs1290-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 9321.959293] LustreError: 280932:0:(fld_handler.c:275:fld_server_lookup()) Skipped 5228 previous similar messages [ 9321.973040] LustreError: 280932:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) fs1290-MDT0001: Trash dir setup failed: rc = -5 [ 9321.978133] LustreError: 280932:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 5339 previous similar messages [ 9322.955082] LustreError: 280932:0:(fld_handler.c:275:fld_server_lookup()) srv-fs1290-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 9322.961042] LustreError: 280932:0:(fld_handler.c:275:fld_server_lookup()) Skipped 11770 previous similar messages [ 9322.973053] LustreError: 280932:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) fs1290-MDT0001: Trash dir setup failed: rc = -5 [ 9322.977098] LustreError: 280932:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 11763 previous similar messages [ 9323.251686] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9324.955076] LustreError: 280932:0:(fld_handler.c:275:fld_server_lookup()) srv-fs1290-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 9324.959266] LustreError: 280932:0:(fld_handler.c:275:fld_server_lookup()) Skipped 24185 previous similar messages [ 9324.973046] LustreError: 280932:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) fs1290-MDT0001: Trash dir setup failed: rc = -5 [ 9324.977182] LustreError: 280932:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 24201 previous similar messages [ 9327.167813] Lustre: server umount fs1290-MDT0001 complete [ 9332.546925] loop1: detected capacity change from 0 to 204800000 [ 9334.860362] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9335.000768] Lustre: 282479:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1290-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9335.006224] LustreError: 282479:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1290-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9335.012417] LustreError: 282481:0:(fld_handler.c:275:fld_server_lookup()) srv-fs1290-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 9335.016859] LustreError: 282481:0:(fld_handler.c:275:fld_server_lookup()) Skipped 22566 previous similar messages [ 9335.021831] LustreError: 282481:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) fs1290-MDT0001: Trash dir setup failed: rc = -5 [ 9335.027054] LustreError: 282481:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 22446 previous similar messages [ 9336.900029] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9339.286300] Lustre: server umount fs1290-MDT0001 complete [ 9349.417825] Lustre: server umount lustre-MDT0000 complete [ 9352.897544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9352.973062] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9354.725527] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9357.807644] loop1: detected capacity change from 0 to 204800000 [ 9358.313041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9358.317107] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 9360.212031] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9360.336391] Lustre: 284890:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1290-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9360.340433] LustreError: 284890:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1290-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9360.345594] LustreError: 284892:0:(fld_handler.c:275:fld_server_lookup()) srv-fs1290-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 9360.350069] LustreError: 284892:0:(fld_handler.c:275:fld_server_lookup()) Skipped 46818 previous similar messages [ 9360.353582] LustreError: 284892:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) fs1290-MDT0001: Trash dir setup failed: rc = -5 [ 9360.357692] LustreError: 284892:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 46818 previous similar messages [ 9361.976853] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9364.070428] Lustre: server umount fs1290-MDT0001 complete [ 9373.672086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9373.673298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9373.679451] Lustre: Skipped 3 previous similar messages [ 9373.684880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9373.691596] Lustre: Skipped 12 previous similar messages [ 9378.209759] Lustre: server umount lustre-OST0000 complete [ 9383.906596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9394.144113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9394.205483] Lustre: server umount lustre-MDT0000 complete [ 9395.645269] LustreError: 276994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346174 with bad export cookie 15308957485250318064 [ 9395.648788] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9395.649234] LustreError: 276994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9395.785185] Lustre: server umount lustre-MDT0001 complete [ 9399.836298] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 9400.957099] Key type lgssc unregistered [ 9401.089438] LNet: 287197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9401.093239] LNetError: 287197:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9401.105342] LNet: Removed LNI 192.168.204.159@tcp [ 9401.421204] Key type .llcrypt unregistered [ 9401.422529] Key type ._llcrypt unregistered [ 9410.438141] Key type ._llcrypt registered [ 9410.439516] Key type .llcrypt registered [ 9410.491607] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 9410.863763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9410.921435] alg: No test for adler32 (adler32-zlib) [ 9411.807460] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 9411.904473] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 9413.496211] Key type lgssc registered [ 9413.936773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9416.900477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9416.908226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9418.136673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9419.810404] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9423.380985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9423.584659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9425.053276] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9427.071194] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9428.430588] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9430.945276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9431.093770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9433.027670] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9433.125049] LustreError: 290401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9433.133905] LustreError: 290401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9434.992668] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9438.181079] LustreError: 290034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9443.303546] LustreError: 290034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9443.310729] LustreError: 290034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9445.646304] LustreError: 290594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9450.761981] LustreError: 290034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9450.768542] LustreError: 290034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9452.018541] loop1: detected capacity change from 0 to 204800000 [ 9452.093400] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9452.097645] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9452.102703] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9453.905242] loop1: detected capacity change from 0 to 204800000 [ 9455.908112] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9455.958694] Lustre: fs19578-OST0000: new disk, initializing [ 9455.960713] Lustre: srv-fs19578-OST0000: No data found on store. Initialize space. [ 9455.963611] Lustre: 291896:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9455.988856] Lustre: fs19578-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9458.049460] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9461.001581] LustreError: 290401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9461.009077] LustreError: 290401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9461.755311] Lustre: server umount fs19578-OST0000 complete [ 9466.006231] loop1: detected capacity change from 0 to 204800000 [ 9467.920682] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9467.980245] Lustre: fs19578-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9469.925137] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9472.827995] loop2: detected capacity change from 0 to 204800000 [ 9472.909289] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9472.914304] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9472.918810] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9474.667826] loop2: detected capacity change from 0 to 204800000 [ 9476.520394] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9476.550165] LustreError: fs19578-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 9476.554400] LustreError: fs19578-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9476.557474] LustreError: 294897:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -13 [ 9476.559870] LustreError: 294897:0:(tgt_mount.c:2084:server_put_super()) no obd fs19578-OST0001 [ 9476.561832] LustreError: 294897:0:(tgt_mount.c:116:server_deregister_mount()) fs19578-OST0001 not registered [ 9476.572643] Lustre: server umount fs19578-OST0001 complete [ 9476.574033] LustreError: 294897:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9479.142243] LustreError: 290031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9479.149057] LustreError: 290031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 9480.261323] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9480.304631] Lustre: fs19578-OST0001: new disk, initializing [ 9480.306638] Lustre: srv-fs19578-OST0001: No data found on store. Initialize space. [ 9480.309678] Lustre: 295447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9480.333134] Lustre: fs19578-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9482.383598] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9486.025906] Lustre: server umount fs19578-OST0000 complete [ 9489.217991] Lustre: server umount fs19578-OST0001 complete [ 9493.185886] loop1: detected capacity change from 0 to 204800000 [ 9495.046282] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9496.933462] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9499.216837] loop2: detected capacity change from 0 to 204800000 [ 9501.264850] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9501.331455] Lustre: fs19578-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9501.333990] Lustre: Skipped 1 previous similar message [ 9503.339642] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9506.498683] Lustre: server umount fs19578-OST0000 complete [ 9509.762552] Lustre: server umount fs19578-OST0001 complete [ 9512.200335] LustreError: 290033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9512.206735] LustreError: 290033:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 9514.978436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9514.980128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9514.982033] LustreError: Skipped 1 previous similar message [ 9514.987199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9514.990512] Lustre: Skipped 1 previous similar message [ 9519.631812] Lustre: server umount lustre-OST0000 complete [ 9521.121061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9521.123878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9521.128467] Lustre: Skipped 1 previous similar message [ 9521.130535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9525.217095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9525.217612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9527.064091] Lustre: server umount lustre-MDT0000 complete [ 9528.274112] LustreError: 288412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346307 with bad export cookie 7798362192845881924 [ 9528.277253] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9528.278036] LustreError: 288412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9528.395399] Lustre: server umount lustre-MDT0001 complete [ 9531.886830] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 9532.932238] Key type lgssc unregistered [ 9533.066349] LNet: 301286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9533.070063] LNetError: 301286:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9533.079326] LNet: Removed LNI 192.168.204.159@tcp [ 9533.361107] Key type .llcrypt unregistered [ 9533.362121] Key type ._llcrypt unregistered [ 9543.729429] Key type ._llcrypt registered [ 9543.730687] Key type .llcrypt registered [ 9543.769343] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 9549.843528] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 9550.238207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9550.265609] alg: No test for adler32 (adler32-zlib) [ 9551.111176] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 9551.190610] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 9552.776215] Key type lgssc registered [ 9553.173048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9564.871359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9564.879575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9565.964491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9565.978663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9566.019967] Lustre: lustre-MDT0000: new disk, initializing [ 9566.076170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9566.083947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9567.406659] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9571.520067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9571.559675] Lustre: 305081:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9571.580330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9571.582242] Lustre: Skipped 1 previous similar message [ 9571.616675] Lustre: lustre-MDT0001: new disk, initializing [ 9571.645850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9571.658043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9571.661654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9572.954472] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9576.071312] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9577.290550] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9579.615896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9579.711849] Lustre: lustre-OST0000: new disk, initializing [ 9579.713960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9579.716926] Lustre: 306825:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9579.747312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9581.627257] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9584.123129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9584.126329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9584.162422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9584.690370] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9585.405413] Lustre: Mounted lustre-client [ 9585.700495] LustreError: 307576:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9a2227a12000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 9585.724636] Lustre: Unmounted lustre-client [ 9589.217031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9589.217221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9589.219465] LustreError: Skipped 1 previous similar message [ 9589.219907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9589.225171] Lustre: Skipped 1 previous similar message [ 9590.736730] Lustre: server umount lustre-OST0000 complete [ 9592.289041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9592.294067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9592.302125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9592.304401] Lustre: Skipped 1 previous similar message [ 9595.873471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9595.874118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9598.110798] Lustre: server umount lustre-MDT0000 complete [ 9599.354354] LustreError: 305860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346378 with bad export cookie 5321792564132135672 [ 9599.358589] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9599.359023] LustreError: 305860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9599.472363] Lustre: server umount lustre-MDT0001 complete [ 9602.982591] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 9604.008161] Key type lgssc unregistered [ 9644.064337] LNet: 308853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9644.067249] LNetError: 308853:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9716.774309] LNet: Removed LNI 192.168.204.159@tcp [ 9717.070073] Key type .llcrypt unregistered [ 9717.071435] Key type ._llcrypt unregistered [ 9722.766464] Key type ._llcrypt registered [ 9722.768068] Key type .llcrypt registered [ 9722.804921] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 9730.769621] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 9731.369820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9731.447786] alg: No test for adler32 (adler32-zlib) [ 9732.415944] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 9732.581949] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 9734.192224] Key type lgssc registered [ 9735.060812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9754.356428] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:42:12 (1780346532) [ 9758.054522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9758.078335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9759.255532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9759.277725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9759.332448] Lustre: lustre-MDT0000: new disk, initializing [ 9759.380046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9759.389816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9761.957334] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9769.113558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9769.195654] Lustre: 312023:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9769.236483] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9769.238957] Lustre: Skipped 1 previous similar message [ 9769.286426] Lustre: lustre-MDT0001: new disk, initializing [ 9769.359673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9769.395979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9769.404451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9772.244960] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9777.271447] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9779.404837] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9783.003808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9783.184977] Lustre: lustre-OST0000: new disk, initializing [ 9783.187538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9783.191661] Lustre: 313764:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9783.252281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9792.129979] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9792.654377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9792.694440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9792.957903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9803.474100] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9824.230153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9824.242508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9824.259431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9824.277220] Lustre: Skipped 1 previous similar message [ 9829.349174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9829.365496] Lustre: Skipped 1 previous similar message [ 9834.476033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9834.504435] Lustre: Skipped 1 previous similar message [ 9834.977520] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9835.323588] Lustre: server umount lustre-OST0000 complete [ 9841.634774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9841.638753] LustreError: Skipped 1 previous similar message [ 9841.641158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9841.647772] Lustre: Skipped 1 previous similar message [ 9841.651281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9844.709319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9847.879336] Lustre: server umount lustre-MDT0000 complete [ 9851.432257] LustreError: 312016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346630 with bad export cookie 3947821476992452794 [ 9851.433556] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9851.452544] LustreError: 312016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9851.752559] Lustre: server umount lustre-MDT0001 complete [ 9860.587031] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 9862.839207] Key type lgssc unregistered [ 9863.067690] LNet: 315426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9863.074383] LNetError: 315426:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9863.089075] LNet: Removed LNI 192.168.204.159@tcp [ 9863.701578] Key type .llcrypt unregistered [ 9863.705065] Key type ._llcrypt unregistered [ 9884.775585] Key type ._llcrypt registered [ 9884.780801] Key type .llcrypt registered [ 9884.875820] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 9885.656722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9885.713079] alg: No test for adler32 (adler32-zlib) [ 9886.702583] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 9886.968655] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 9888.656195] Key type lgssc registered [ 9889.760200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9895.770366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9895.789795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9897.306645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9901.225859] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9907.683652] LustreError: 316831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9907.701692] LustreError: 316831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9908.464870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9908.720503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9912.907570] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9917.953270] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9921.584566] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9927.975096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9928.208625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9934.169204] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9939.794096] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9952.427314] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9953.616357] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9958.884291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9958.889378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9958.892769] Lustre: Skipped 2 previous similar messages [ 9958.899758] Lustre: Skipped 1 previous similar message [ 9961.777622] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.59@tcp (stopping) [ 9961.784429] Lustre: Skipped 1 previous similar message [ 9964.002670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9964.007133] Lustre: Skipped 2 previous similar messages [ 9966.857856] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.59@tcp (stopping) [ 9971.168140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9971.367242] Lustre: server umount lustre-MDT0000 complete [ 9971.993743] LustreError: 316827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9974.761789] LustreError: 316813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346753 with bad export cookie 17720908211161534522 [ 9974.771323] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9974.772542] LustreError: 316813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9977.101032] LustreError: 317853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9977.117448] LustreError: 317853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9977.123501] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.59@tcp (stopping) [ 9977.130728] Lustre: Skipped 3 previous similar messages [ 9977.316801] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9979.368919] LustreError: 316828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9979.387377] LustreError: 316828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9984.480735] LustreError: 317853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9984.509421] LustreError: 317853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9987.338276] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.59@tcp (stopping) [ 9987.341134] Lustre: Skipped 4 previous similar messages [ 9989.088160] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9989.288952] Lustre: server umount lustre-MDT0001 complete [ 9992.884455] Lustre: server umount lustre-OST0000 complete [10000.338140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10003.990813] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10006.685664] Lustre: 320560:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.159@tcp [10008.600956] Lustre: server umount MGS complete [10015.599868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10015.938246] LustreError: 321131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10015.953164] LustreError: 321131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10016.012377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10019.605427] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10026.745128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10027.017566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10030.640848] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10038.211574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10038.493129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10044.295759] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10049.573557] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10050.988251] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10054.552430] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10056.081749] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10061.285312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10061.303744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10061.327737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10061.337014] Lustre: Skipped 1 previous similar message [10062.315830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10062.331064] LustreError: Skipped 1 previous similar message [10066.423712] Lustre: server umount lustre-OST0000 complete [10072.545611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10072.554918] Lustre: Skipped 1 previous similar message [10075.815661] Lustre: server umount lustre-MDT0000 complete [10077.664756] LustreError: 321131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10077.675164] LustreError: 321131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10079.206803] LustreError: 321112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346858 with bad export cookie 17720908211161535362 [10079.207765] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10079.223309] LustreError: 321112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10079.673231] Lustre: server umount lustre-MDT0001 complete [10088.969870] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10091.438931] Key type lgssc unregistered [10091.689720] LNet: 324307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10091.703266] LNetError: 324307:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10091.734469] LNet: Removed LNI 192.168.204.159@tcp [10092.614850] Key type .llcrypt unregistered [10092.617526] Key type ._llcrypt unregistered [10108.978898] Key type ._llcrypt registered [10108.980863] Key type .llcrypt registered [10109.080900] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [10123.053978] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [10124.142852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10124.165902] alg: No test for adler32 (adler32-zlib) [10125.225285] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [10125.439942] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [10127.152760] Key type lgssc registered [10128.353672] Lustre: Echo OBD driver; http://www.lustre.org/ [10160.635170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10160.657297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10161.932229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10161.959804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10162.041642] Lustre: lustre-MDT0000: new disk, initializing [10162.166402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10162.189959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10166.264738] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10175.844205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10175.952359] Lustre: 327437:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10175.998290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10176.001049] Lustre: Skipped 1 previous similar message [10176.067894] Lustre: lustre-MDT0001: new disk, initializing [10176.199332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10176.247278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10176.270595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10179.870800] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10186.415953] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10189.501909] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10195.189804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10195.474289] Lustre: lustre-OST0000: new disk, initializing [10195.477541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10195.484414] Lustre: 329181:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10195.566444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10201.088990] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10205.266342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10205.277352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10205.401678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10207.783584] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10212.719758] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10212.975523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10217.444601] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10217.634726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10221.031682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.038287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10221.038604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10221.048082] Lustre: Skipped 1 previous similar message [10221.068422] Lustre: Skipped 1 previous similar message [10225.445486] Lustre: server umount lustre-OST0000 complete [10231.267697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10231.270758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10231.280282] Lustre: Skipped 1 previous similar message [10231.292454] Lustre: Skipped 1 previous similar message [10234.899707] Lustre: server umount lustre-MDT0000 complete [10237.879790] LustreError: 327428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780347016 with bad export cookie 1171265196512948168 [10237.885789] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10237.889423] LustreError: 327428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10238.162919] Lustre: server umount lustre-MDT0001 complete [10245.701864] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 16:50:23 (1780347023) [10259.568088] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [10267.919896] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [10296.883707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10297.075564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10297.110342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10297.162789] Lustre: lustre-MDT0000: new disk, initializing [10297.212280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10297.231986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10300.266334] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10308.309769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10308.402487] Lustre: 333754:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10308.444445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10308.453348] Lustre: Skipped 1 previous similar message [10308.498875] Lustre: lustre-MDT0001: new disk, initializing [10308.562269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10308.592223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10308.602171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10311.416476] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10316.766270] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10319.328272] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10323.790769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10323.962613] Lustre: lustre-OST0000: new disk, initializing [10323.965628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10323.970509] Lustre: 335497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10324.017241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10327.957568] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10332.728475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10332.743375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10332.847454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10334.173856] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10358.755856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10358.773658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10358.776136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10358.780551] LustreError: Skipped 1 previous similar message [10358.791417] Lustre: Skipped 1 previous similar message [10363.873332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10363.889969] Lustre: Skipped 1 previous similar message [10364.526543] Lustre: server umount lustre-OST0000 complete [10368.994377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10368.998232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10369.008620] Lustre: Skipped 2 previous similar messages [10369.025097] Lustre: Skipped 1 previous similar message [10379.233962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10379.247809] Lustre: Skipped 3 previous similar messages [10382.304659] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10382.475097] Lustre: server umount lustre-MDT0000 complete [10384.352908] LustreError: 334994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10384.362520] LustreError: 334994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10385.207825] LustreError: 333746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780347164 with bad export cookie 1171265196512949085 [10385.212347] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10385.223555] LustreError: 333746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10385.561420] Lustre: server umount lustre-MDT0001 complete [10393.274920] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10395.447534] Key type lgssc unregistered [10395.670800] LNet: 337519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10395.680962] LNetError: 337519:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10396.710582] LNet: Removed LNI 192.168.204.159@tcp [10397.284269] Key type .llcrypt unregistered [10397.286347] Key type ._llcrypt unregistered [10460.729876] Key type ._llcrypt registered [10460.737403] Key type .llcrypt registered [10460.833195] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10461.065468] Key type .llcrypt unregistered [10461.068309] Key type ._llcrypt unregistered [10513.991737] Key type ._llcrypt registered [10513.993471] Key type .llcrypt registered [10514.091936] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10514.326298] Key type .llcrypt unregistered [10514.328181] Key type ._llcrypt unregistered [10536.523431] Key type ._llcrypt registered [10536.526054] Key type .llcrypt registered [10536.641573] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10536.860499] Key type .llcrypt unregistered [10536.866573] Key type ._llcrypt unregistered [10557.658800] Key type ._llcrypt registered [10557.663497] Key type .llcrypt registered [10557.759887] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10557.979479] Key type .llcrypt unregistered [10557.983861] Key type ._llcrypt unregistered [10612.007487] Key type ._llcrypt registered [10612.009299] Key type .llcrypt registered [10612.094275] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10612.294307] Key type .llcrypt unregistered [10612.296129] Key type ._llcrypt unregistered [10631.912806] Key type ._llcrypt registered [10631.914721] Key type .llcrypt registered [10632.008796] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10632.231405] Key type .llcrypt unregistered [10632.233415] Key type ._llcrypt unregistered [10650.194332] Key type ._llcrypt registered [10650.196592] Key type .llcrypt registered [10650.252449] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10650.419519] Key type .llcrypt unregistered [10650.421645] Key type ._llcrypt unregistered [10670.108762] Key type ._llcrypt registered [10670.115056] Key type .llcrypt registered [10670.193847] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10670.399396] Key type .llcrypt unregistered [10670.404162] Key type ._llcrypt unregistered [10690.623631] Key type ._llcrypt registered [10690.627665] Key type .llcrypt registered [10690.726889] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10690.979294] Key type .llcrypt unregistered [10690.982848] Key type ._llcrypt unregistered [10712.835551] Key type ._llcrypt registered [10712.837735] Key type .llcrypt registered [10712.930040] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10713.182753] Key type .llcrypt unregistered [10713.187218] Key type ._llcrypt unregistered [10754.682686] Key type ._llcrypt registered [10754.689288] Key type .llcrypt registered [10754.808059] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10755.083648] Key type .llcrypt unregistered [10755.089538] Key type ._llcrypt unregistered [10776.067142] Key type ._llcrypt registered [10776.070899] Key type .llcrypt registered [10776.170434] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10776.423673] Key type .llcrypt unregistered [10776.427255] Key type ._llcrypt unregistered [10796.032820] Key type ._llcrypt registered [10796.034549] Key type .llcrypt registered [10796.097443] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10796.261519] Key type .llcrypt unregistered [10796.266889] Key type ._llcrypt unregistered [10832.814629] Key type ._llcrypt registered [10832.821592] Key type .llcrypt registered [10832.918971] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [10844.447884] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [10845.083768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10845.141140] alg: No test for adler32 (adler32-zlib) [10846.175240] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [10846.369731] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [10848.032201] Key type lgssc registered [10848.775682] Lustre: Echo OBD driver; http://www.lustre.org/ [10872.782158] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10872.808186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10873.993976] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10874.027474] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10874.126840] Lustre: tst250fs-MDT0000: new disk, initializing [10874.200945] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10874.212530] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10877.829728] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10885.035808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10885.115380] Lustre: 357203:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10885.169771] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10885.173322] Lustre: Skipped 1 previous similar message [10885.218608] Lustre: tst250fs-MDT0001: new disk, initializing [10885.297823] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10885.335513] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10885.347781] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10887.795037] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10891.021492] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10894.888291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10895.066230] Lustre: tst250fs-OST0000: new disk, initializing [10895.073768] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10895.080702] Lustre: 358814:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10895.143528] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10897.118506] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10897.125753] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10897.220615] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10898.809669] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10905.734501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10905.848380] Lustre: tst250fs-OST0001: new disk, initializing [10905.851462] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10905.855818] Lustre: 359668:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10905.921832] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10909.947343] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10914.154886] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10914.966424] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10914.973602] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10915.084706] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10930.661166] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10930.662271] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10930.674262] Lustre: Skipped 3 previous similar messages [10930.680595] Lustre: Skipped 2 previous similar messages [10935.337482] Lustre: server umount tst250fs-MDT0000 complete [10937.932940] LustreError: 357196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780347716 with bad export cookie 11666177202373459842 [10937.935911] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10937.940380] LustreError: 357196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10938.176071] Lustre: server umount tst250fs-MDT0001 complete [10950.176728] Lustre: server umount tst250fs-OST0000 complete [10962.758389] Lustre: server umount tst250fs-OST0001 complete [10974.900200] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [10980.044098] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [11001.047548] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 17:02:59 (1780347779) [11002.119744] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [11012.892466] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [11018.480291] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [11036.774889] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10724 sec ========= 17:03:35 (1780347815) [11037.666464] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:03:36 (1780347816) === [11039.637846] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:03:38 (1780347818) === [11063.794397] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [11065.510646] Key type lgssc unregistered [11065.714852] LNet: 368700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11065.722412] LNetError: 368700:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11065.737361] LNet: Removed LNI 192.168.204.159@tcp [11066.364167] Key type .llcrypt unregistered [11066.367802] Key type ._llcrypt unregistered