[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 500187214 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2448MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002278] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004025] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013078] *** VALIDATE selinux *** [ 0.021786] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026760] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029137] *** VALIDATE tmpfs *** [ 0.030478] *** VALIDATE proc *** [ 0.032128] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034314] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036090] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037059] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038044] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042706] debug: unmapping init [mem 0xffffffffbb059000-0xffffffffbb060fff] [ 0.044175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045731] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046022] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053331] rcu: Hierarchical SRCU implementation. [ 0.055613] smp: Bringing up secondary CPUs ... [ 0.056607] x86: Booting SMP configuration: [ 0.057028] .... node #0, CPUs: #1 #2 #3 [ 0.060564] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.132116] node 0 deferred pages initialised in 67ms [ 0.137032] devtmpfs: initialized [ 0.138331] x86/mm: Memory block size: 128MB [ 0.140917] gcov: version magic: 0x41383552 [ 0.144290] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150314] pinctrl core: initialized pinctrl subsystem [ 0.152202] [ 0.152764] ************************************************************* [ 0.155013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157010] ** ** [ 0.159013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161014] ** ** [ 0.163014] ** This means that this kernel is built to expose internal ** [ 0.165014] ** IOMMU data structures, which may compromise security on ** [ 0.167014] ** your system. ** [ 0.169027] ** ** [ 0.171013] ** If you see this message and you are not debugging the ** [ 0.173011] ** kernel, report this immediately to your vendor! ** [ 0.175014] ** ** [ 0.177012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179014] ************************************************************* [ 0.181765] NET: Registered protocol family 16 [ 0.184470] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.186062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.194040] cpuidle: using governor menu [ 0.196075] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.197511] PCI: Using configuration type 1 for base access [ 0.199141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209145] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.211080] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.215089] cryptd: max_cpu_qlen set to 1000 [ 0.218216] ACPI: Added _OSI(Module Device) [ 0.220016] ACPI: Added _OSI(Processor Device) [ 0.221012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.223012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228641] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235290] ACPI: Interpreter enabled [ 0.236055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237011] ACPI: Using IOAPIC for interrupt routing [ 0.239116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.242361] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.252025] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.254054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.257019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.261078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.267559] acpiphp: Slot [2] registered [ 0.268156] acpiphp: Slot [5] registered [ 0.269263] acpiphp: Slot [6] registered [ 0.271102] acpiphp: Slot [7] registered [ 0.272095] acpiphp: Slot [8] registered [ 0.273074] acpiphp: Slot [9] registered [ 0.274095] acpiphp: Slot [10] registered [ 0.275077] acpiphp: Slot [3] registered [ 0.277063] acpiphp: Slot [4] registered [ 0.278080] acpiphp: Slot [11] registered [ 0.280123] acpiphp: Slot [12] registered [ 0.281083] acpiphp: Slot [13] registered [ 0.282085] acpiphp: Slot [14] registered [ 0.285129] acpiphp: Slot [15] registered [ 0.286076] acpiphp: Slot [16] registered [ 0.287074] acpiphp: Slot [17] registered [ 0.289085] acpiphp: Slot [18] registered [ 0.291126] acpiphp: Slot [19] registered [ 0.293116] acpiphp: Slot [20] registered [ 0.294105] acpiphp: Slot [21] registered [ 0.296114] acpiphp: Slot [22] registered [ 0.298110] acpiphp: Slot [23] registered [ 0.299118] acpiphp: Slot [24] registered [ 0.301111] acpiphp: Slot [25] registered [ 0.303115] acpiphp: Slot [26] registered [ 0.304122] acpiphp: Slot [27] registered [ 0.306132] acpiphp: Slot [28] registered [ 0.307079] acpiphp: Slot [29] registered [ 0.308086] acpiphp: Slot [30] registered [ 0.310109] acpiphp: Slot [31] registered [ 0.311067] PCI host bridge to bus 0000:00 [ 0.313024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.315024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.318021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.321023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.323020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.327032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329252] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.333102] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.336395] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.347018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.351965] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.355019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.358023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.361020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.364534] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.367836] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.373762] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.377018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.390017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.395000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.401376] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.408019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.418016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.437021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.448354] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.464030] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.472015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.501014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.509915] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.515017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.523015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.546014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.559227] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.567016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.572015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.592016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.603872] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.613011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.619015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.637017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.648815] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.654019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.659013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.684076] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.699063] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.701389] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.704705] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.707549] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.710261] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.714125] iommu: Default domain type: Passthrough [ 0.716420] SCSI subsystem initialized [ 0.718144] ACPI: bus type USB registered [ 0.720109] usbcore: registered new interface driver usbfs [ 0.722066] usbcore: registered new interface driver hub [ 0.723147] usbcore: registered new device driver usb [ 0.725207] pps_core: LinuxPPS API ver. 1 registered [ 0.727010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.730082] PTP clock support registered [ 0.732078] EDAC MC: Ver: 3.0.0 [ 0.735145] PCI: Using ACPI for IRQ routing [ 0.737388] NetLabel: Initializing [ 0.738010] NetLabel: domain hash size = 128 [ 0.739010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.741085] NetLabel: unlabeled traffic allowed by default [ 0.744053] vgaarb: loaded [ 0.745190] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.748015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.754011] clocksource: Switched to clocksource kvm-clock [ 0.858099] VFS: Disk quotas dquot_6.6.0 [ 0.860381] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.863238] *** VALIDATE ramfs *** [ 0.864415] *** VALIDATE hugetlbfs *** [ 0.866343] pnp: PnP ACPI init [ 0.869486] pnp: PnP ACPI: found 6 devices [ 0.892862] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.897154] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.898952] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.901218] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.903802] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.906409] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.909470] NET: Registered protocol family 2 [ 0.911800] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.917080] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.921899] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.927127] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.930802] TCP: Hash tables configured (established 65536 bind 65536) [ 0.932944] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.935369] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.938248] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.940552] NET: Registered protocol family 1 [ 0.943216] RPC: Registered named UNIX socket transport module. [ 0.948148] RPC: Registered udp transport module. [ 0.949970] RPC: Registered tcp transport module. [ 0.951824] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.954227] NET: Registered protocol family 44 [ 0.956095] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.958095] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.960387] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.962927] PCI: CLS 0 bytes, default 64 [ 0.964611] Unpacking initramfs... [ 2.371545] debug: unmapping init [mem 0xffff9bca7cc54000-0xffff9bca7ffbffff] [ 2.374498] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.377160] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.380030] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 2.870286] Initialise system trusted keyrings [ 2.872142] Key type blacklist registered [ 2.873990] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.883775] zbud: loaded [ 2.886994] *** VALIDATE nfs *** [ 2.888313] *** VALIDATE nfs4 *** [ 2.890096] pstore: using deflate compression [ 2.893881] Platform Keyring initialized [ 2.988509] NET: Registered protocol family 38 [ 2.990364] Key type asymmetric registered [ 2.991618] Asymmetric key parser 'x509' registered [ 2.993142] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.996043] io scheduler mq-deadline registered [ 2.997446] io scheduler kyber registered [ 3.000022] io scheduler bfq registered [ 3.001684] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.004117] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.006551] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.009146] ACPI: Power Button [PWRF] [ 3.015079] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.021911] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.035648] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.043787] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.073564] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.104708] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.132809] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.137321] Non-volatile memory driver v1.3 [ 3.139141] Linux agpgart interface v0.103 [ 3.174839] virtio_blk virtio1: [vda] 135992 512-byte logical blocks (69.6 MB/66.4 MiB) [ 3.177827] vda: detected capacity change from 0 to 69627904 [ 3.192794] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.195797] vdb: detected capacity change from 0 to 1073741824 [ 3.216118] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.219485] vdc: detected capacity change from 0 to 2621440000 [ 3.234878] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.238505] vdd: detected capacity change from 0 to 2621440000 [ 3.255064] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.257261] vde: detected capacity change from 0 to 4294967296 [ 3.271827] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.273983] vdf: detected capacity change from 0 to 4294967296 [ 3.283274] libphy: Fixed MDIO Bus: probed [ 3.289651] usbcore: registered new interface driver usbserial_generic [ 3.291668] usbserial: USB Serial support registered for generic [ 3.293836] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.298503] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.299840] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.302380] mousedev: PS/2 mouse device common for all mice [ 3.304976] rtc_cmos 00:05: RTC can wake from S4 [ 3.307326] rtc_cmos 00:05: registered as rtc0 [ 3.308994] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.311203] intel_pstate: CPU model not supported [ 3.313679] hid: raw HID events driver (C) Jiri Kosina [ 3.315244] usbcore: registered new interface driver usbhid [ 3.316856] usbhid: USB HID core driver [ 3.318382] drop_monitor: Initializing network drop monitor service [ 3.318418] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.320325] Initializing XFRM netlink socket [ 3.326103] NET: Registered protocol family 10 [ 3.328935] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.331124] Segment Routing with IPv6 [ 3.337228] NET: Registered protocol family 17 [ 3.338314] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.339375] mpls_gso: MPLS GSO support [ 3.352524] RAS: Correctable Errors collector initialized. [ 3.354773] AVX version of gcm_enc/dec engaged. [ 3.356660] AES CTR mode by8 optimization enabled [ 3.423667] sched_clock: Marking stable (3423636958, 0)->(4338268766, -914631808) [ 3.427305] registered taskstats version 1 [ 3.430535] Loading compiled-in X.509 certificates [ 3.432274] zswap: loaded using pool lzo/zbud [ 3.456024] Key type big_key registered [ 3.466705] Key type encrypted registered [ 3.469066] ima: No TPM chip found, activating TPM-bypass! [ 3.471325] ima: Allocated hash algorithm: sha1 [ 3.473258] ima: No architecture policies found [ 3.479309] evm: Initialising EVM extended attributes: [ 3.481508] evm: security.selinux [ 3.483039] evm: security.ima [ 3.484336] evm: security.capability [ 3.485905] evm: HMAC attrs: 0x1 [ 3.490177] rtc_cmos 00:05: setting system clock to 2026-06-07 17:15:23 UTC (1780852523) [ 3.497476] debug: unmapping init [mem 0xffffffffbc003000-0xffffffffbc1fffff] [ 3.501219] debug: unmapping init [mem 0xffffffffbad82000-0xffffffffbb058fff] [ 3.511146] Write protecting the kernel read-only data: 28672k [ 3.515912] debug: unmapping init [mem 0xffffffffb9403000-0xffffffffb95fffff] [ 3.519814] debug: unmapping init [mem 0xffffffffb9d14000-0xffffffffb9dfffff] [ 3.558515] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.568454] systemd[1]: Detected virtualization kvm. [ 3.574159] systemd[1]: Detected architecture x86-64. [ 3.576142] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.604354] systemd[1]: No hostname configured. [ 3.606372] systemd[1]: Set hostname to . [ 3.608847] random: systemd: uninitialized urandom read (16 bytes read) [ 3.611966] systemd[1]: Initializing machine ID from random generator. [ 3.674492] random: ln: uninitialized urandom read (6 bytes read) [ 3.762734] random: systemd: uninitialized urandom read (16 bytes read) [ 3.765454] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.770096] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.775572] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Timers. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.492138] device-mapper: uevent: version 1.0.3 [ 4.499925] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.283143] virtio_net virtio0 ens2: renamed from eth0 [ 5.312406] random: fast init done [ 5.356532] scsi host0: ata_piix [ 5.362965] scsi host1: ata_piix [ 5.364771] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.367321] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.998930] random: crng init done [ 10.002100] random: 7 urandom warning(s) missed due to ratelimiting [ 10.126550] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.696754] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.251189] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.926325] SELinux: Disabled at runtime. [ 15.048709] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.060896] systemd[1]: Detected virtualization kvm. [ 15.062887] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.333918] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.341081] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.350259] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.369343] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.381222] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.401529] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.410668] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 16.901487] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 17.811619] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.679741] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.791445] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.510689] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.619664] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 24.897949] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root sup[ 25.936856] NFS: Registering the id_resolver key type port (9s / no limit)[ 25.956215] Key type id_resolver registered [ 25.964033] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg451-server login: [ 91.440932] libcfs: loading out-of-tree module taints kernel. [ 91.510351] Key type ._llcrypt registered [ 91.515111] Key type .llcrypt registered [ 91.603438] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 112.059993] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 113.899377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 113.914958] alg: No test for adler32 (adler32-zlib) [ 115.639407] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 116.694685] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 118.481080] Key type lgssc registered [ 120.791613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 142.321213] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 186.607658] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 199.580916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 199.621958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.945046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 200.991960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 201.070516] Lustre: lustre-MDT0000: new disk, initializing [ 201.205389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.233719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 205.903537] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.449942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 218.587715] Lustre: 6509:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 218.618772] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 218.625354] Lustre: Skipped 1 previous similar message [ 218.711401] Lustre: lustre-MDT0001: new disk, initializing [ 218.785890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 218.814606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 218.821484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 223.065074] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.465095] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 239.687533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.981987] Lustre: lustre-OST0000: new disk, initializing [ 239.989300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 240.009790] Lustre: 8412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.104917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 245.781670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 245.791556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 245.880812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.134441] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.240955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 263.431261] Lustre: lustre-OST0001: new disk, initializing [ 263.435431] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 263.452710] Lustre: 9468:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 263.518703] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 270.991150] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.588078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 273.599993] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 273.670584] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 285.801041] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 293.063326] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 299.229576] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing check_logdir /tmp/testlogs/ [ 304.755904] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing yml_node [ 310.284807] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 313.267504] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 316.189760] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 317.904132] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Jun 7 13:20:36 EDT 2026 [ 334.392101] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 335.581678] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 350.688878] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.691983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 350.704470] Lustre: Skipped 3 previous similar messages [ 350.716232] Lustre: Skipped 2 previous similar messages [ 355.818925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 355.828838] Lustre: Skipped 3 previous similar messages [ 356.738178] Lustre: server umount lustre-MDT0000 complete [ 360.506575] LustreError: 9479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780852880 with bad export cookie 620729543766906074 [ 360.515364] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 360.517253] LustreError: 9479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 360.929943] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.943979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 360.949169] Lustre: Skipped 1 previous similar message [ 366.054680] LustreError: 9478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.054820] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.084770] LustreError: 9478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 366.094148] Lustre: Skipped 1 previous similar message [ 366.963815] Lustre: server umount lustre-MDT0001 complete [ 371.116588] Lustre: server umount lustre-OST0000 complete [ 374.958810] Lustre: server umount lustre-OST0001 complete [ 381.340656] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 389.972521] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 423.058210] hrtimer: interrupt took 3202230 ns [ 425.407639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.616485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 425.655663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 425.781218] Lustre: lustre-MDT0000: new disk, initializing [ 425.902331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.926562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 430.113555] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 440.818952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.954626] Lustre: 14991:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 441.000737] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 441.005407] Lustre: Skipped 1 previous similar message [ 441.089118] Lustre: lustre-MDT0001: new disk, initializing [ 441.174576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 441.211724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 441.215623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 446.110546] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 454.425316] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 458.631828] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 465.105622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.532599] Lustre: lustre-OST0000: new disk, initializing [ 465.539727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 465.557882] Lustre: 16732:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 465.706277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.486587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 467.501548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 467.623080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 472.415652] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 480.350877] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 485.350445] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 485.663300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 490.869503] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 491.074316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 493.553356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 493.556309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.578686] Lustre: Skipped 1 previous similar message [ 493.593394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 493.603642] Lustre: Skipped 1 previous similar message [ 503.779792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.793762] Lustre: Skipped 3 previous similar messages [ 507.360314] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 507.520496] Lustre: server umount lustre-OST0000 complete [ 512.995394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 513.004484] LustreError: Skipped 1 previous similar message [ 513.015095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.028210] Lustre: Skipped 1 previous similar message [ 517.519861] Lustre: server umount lustre-MDT0000 complete [ 519.139426] LustreError: 17925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.154180] LustreError: 17925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 521.655838] LustreError: 14983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780853041 with bad export cookie 620729543766907607 [ 521.668073] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.678802] LustreError: 14983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 522.110960] Lustre: server umount lustre-MDT0001 complete [ 528.180588] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 13:24:06 (1780853046) [ 547.698601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.233575] LustreError: 19464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.302524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.449587] LustreError: 19465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.797843] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 558.563072] LustreError: 19464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.832496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.058784] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 574.694210] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 579.344912] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 581.806453] Lustre: Failing over lustre-MDT0000 [ 582.163308] Lustre: server umount lustre-MDT0000 complete [ 583.648607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 583.653918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.664098] Lustre: Skipped 1 previous similar message [ 583.667802] LustreError: 19464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.547769] Lustre: 3660:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780853104/real 1780853104] req@ffff9bcaf65c7800 x1867359333805952/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780853120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 600.584218] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 600.609813] LustreError: 19464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.630498] LustreError: 19464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 602.624899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.153233] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 615.719643] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 616.423974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 616.425367] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 616.493523] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 624.163533] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 631.780436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 631.785833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 631.788207] Lustre: Skipped 2 previous similar messages [ 631.791443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 631.794356] Lustre: Skipped 4 previous similar messages [ 634.342361] Lustre: server umount lustre-MDT0000 complete [ 637.658448] LustreError: 19444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780853157 with bad export cookie 620729543766909021 [ 637.667427] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 637.671115] LustreError: 19444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 638.124049] Lustre: server umount lustre-MDT0001 complete [ 647.260964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.866473] LustreError: 22465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.881253] LustreError: 22465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 647.949456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 647.952797] Lustre: Skipped 2 previous similar messages [ 652.709927] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 661.340026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.195874] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 673.899351] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 678.417353] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 681.096919] Lustre: Failing over lustre-MDT0000 [ 681.725979] Lustre: server umount lustre-MDT0000 complete [ 682.474761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 682.504984] Lustre: Skipped 1 previous similar message [ 698.783262] Lustre: 3658:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780853202/real 1780853202] req@ffff9bc9c2065f80 x1867359333834624/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780853218 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 698.847789] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 701.784883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 708.528270] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 712.591119] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 713.701921] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 713.705336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 713.730780] Lustre: Skipped 1 previous similar message [ 713.748089] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 720.420851] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 729.058283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 729.059538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.078051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 729.093028] Lustre: Skipped 1 previous similar message [ 729.110547] Lustre: Skipped 3 previous similar messages [ 730.998301] Lustre: server umount lustre-MDT0000 complete [ 731.172339] LustreError: 22466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 731.193098] LustreError: 22466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 734.644218] LustreError: 22447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780853254 with bad export cookie 620729543766909959 [ 734.656023] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 734.662760] LustreError: 22447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 734.957693] Lustre: server umount lustre-MDT0001 complete [ 743.780534] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 13:27:42 (1780853262) [ 758.904510] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 767.686553] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 805.928666] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 817.926649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.298051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 818.330607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 818.420596] Lustre: lustre-MDT0000: new disk, initializing [ 818.521884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 818.530207] Lustre: Skipped 2 previous similar messages [ 818.558465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 823.165373] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 835.304322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 835.453683] Lustre: 28460:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 835.852143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 840.901614] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 849.478530] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 853.938504] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 861.471541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 861.787276] Lustre: 30202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 863.669530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 863.734916] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 863.795962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 863.818540] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 868.898431] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 877.851574] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 884.325180] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 909.802399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 909.802944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.817748] LustreError: Skipped 1 previous similar message [ 909.819699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 909.845806] Lustre: Skipped 1 previous similar message [ 909.875306] Lustre: Skipped 1 previous similar message [ 913.280365] Lustre: server umount lustre-OST0000 complete [ 924.029475] Lustre: server umount lustre-MDT0000 complete [ 925.153652] LustreError: 28466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 925.171761] LustreError: 28466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 927.271303] LustreError: 28452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780853447 with bad export cookie 620729543766910365 [ 927.278987] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 927.281068] LustreError: 28452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 927.743317] Lustre: server umount lustre-MDT0001 complete [ 938.838067] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 941.806795] Key type lgssc unregistered [ 942.120078] LNet: 32053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.131641] LNetError: 32053:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.146945] LNet: Removed LNI 192.168.204.151@tcp [ 942.951218] Key type .llcrypt unregistered [ 942.954911] Key type ._llcrypt unregistered [ 971.470225] Key type ._llcrypt registered [ 971.472025] Key type .llcrypt registered [ 971.591736] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 973.029721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 973.085494] alg: No test for adler32 (adler32-zlib) [ 974.263713] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 974.556866] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 976.271824] Key type lgssc registered [ 977.842972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 985.628421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 985.649348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.273348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.728710] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 997.871949] LustreError: 33450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.900804] LustreError: 33450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 999.910203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1000.288087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1005.301990] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1011.691683] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1015.900866] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1022.628785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1022.916049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1025.001619] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1029.349258] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1032.196877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1036.596613] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1051.661509] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 13:32:49 (1780853569) [ 1053.592398] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1055.764171] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1064.209774] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 13:33:02 (1780853582) [ 1074.765892] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 13:33:13 (1780853593) [ 1075.863170] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1086.225736] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1086.232776] Lustre: Skipped 2 previous similar messages [ 1088.514292] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 13:33:26 (1780853606) [ 1090.784964] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1090.788961] Lustre: Skipped 2 previous similar messages [ 1100.663884] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1100.672502] Lustre: Skipped 4 previous similar messages [ 1115.903270] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 13:33:54 (1780853634) [ 1119.103113] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1122.466747] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1130.985874] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 13:34:09 (1780853649) [ 1132.659993] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1132.667761] Lustre: Skipped 6 previous similar messages [ 1148.667145] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 13:34:26 (1780853666) [ 1165.284177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.288602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1165.300374] Lustre: Skipped 1 previous similar message [ 1165.328850] Lustre: Skipped 1 previous similar message [ 1170.412631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1170.427554] Lustre: Skipped 1 previous similar message [ 1171.036578] Lustre: server umount lustre-OST0000 complete [ 1175.521865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.532206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1180.641058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1180.651126] Lustre: Skipped 2 previous similar messages [ 1181.793831] Lustre: server umount lustre-MDT0000 complete [ 1185.771741] LustreError: 33450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.786614] LustreError: 33450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1185.907722] LustreError: 36673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780853705 with bad export cookie 4698517266105183448 [ 1185.916325] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1185.929530] LustreError: 36673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1186.677958] Lustre: server umount lustre-MDT0001 complete [ 1199.757052] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 1202.825874] Key type lgssc unregistered [ 1203.140072] LNet: 40951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1203.162446] LNetError: 40951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.194131] LNet: Removed LNI 192.168.204.151@tcp [ 1204.023728] Key type .llcrypt unregistered [ 1204.026748] Key type ._llcrypt unregistered [ 1223.999488] Key type ._llcrypt registered [ 1224.003513] Key type .llcrypt registered [ 1224.107809] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 1240.291496] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 1241.432461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1241.445861] alg: No test for adler32 (adler32-zlib) [ 1242.588481] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 1242.813811] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 1244.479181] Key type lgssc registered [ 1245.673514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1286.308294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1286.366458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.690718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1287.735551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1287.827825] Lustre: lustre-MDT0000: new disk, initializing [ 1287.938293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.955853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1292.928647] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1303.926500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1304.095530] Lustre: 44074:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1304.158388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1304.164036] Lustre: Skipped 1 previous similar message [ 1304.259503] Lustre: lustre-MDT0001: new disk, initializing [ 1304.372554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1304.428295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1304.436190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1308.952921] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1317.042993] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1321.198614] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1328.799391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1329.088406] Lustre: lustre-OST0000: new disk, initializing [ 1329.093630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1329.100095] Lustre: 45818:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1329.231358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1335.611983] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1336.377059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1336.385824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1336.534271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1343.724846] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1348.838203] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1349.099677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1353.537914] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1353.755344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1357.286311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.291589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1357.292654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1357.305110] Lustre: Skipped 1 previous similar message [ 1357.309849] LustreError: Skipped 1 previous similar message [ 1362.407193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1362.415781] Lustre: Skipped 2 previous similar messages [ 1367.524271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1367.535976] Lustre: Skipped 1 previous similar message [ 1370.080182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1370.347720] Lustre: server umount lustre-OST0000 complete [ 1376.226280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1376.238651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.255093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1377.771733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.628446] Lustre: server umount lustre-MDT0000 complete [ 1385.511914] LustreError: 44862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780853905 with bad export cookie 10245820096745341610 [ 1385.520344] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1386.118046] Lustre: server umount lustre-MDT0001 complete [ 1397.848480] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:38:36 (1780853916) [ 1406.093746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.670425] LustreError: 48135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1406.700757] LustreError: 48135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1406.766863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.768550] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1412.070394] LustreError: 48136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1416.172069] LustreError: 48135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1421.283375] LustreError: 48136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1421.427041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.940365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1426.349706] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1432.642198] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1436.875908] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1443.833676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1444.235669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1450.183879] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1456.105667] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1462.344192] LustreError: 50323:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1463.470571] Lustre: Modifying parameter general.timeout=1 in log params [ 1464.601912] Lustre: Modifying parameter general.timeout=2 in log params [ 1467.731687] Lustre: Modifying parameter general.timeout=5 in log params [ 1467.738719] Lustre: Skipped 2 previous similar messages [ 1472.556060] Lustre: Modifying parameter general.timeout=10 in log params [ 1472.565809] Lustre: Skipped 4 previous similar messages [ 1481.181030] Lustre: Modifying parameter general.timeout=19 in log params [ 1481.191548] Lustre: Skipped 8 previous similar messages [ 1497.685586] Lustre: Modifying parameter general.timeout=37 in log params [ 1497.691489] Lustre: Skipped 17 previous similar messages [ 1530.085788] Lustre: Modifying parameter general.timeout=77 in log params [ 1530.090113] Lustre: Skipped 39 previous similar messages [ 1587.600580] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 13:41:46 (1780854106) [ 1590.261983] Lustre: 57268:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1591.920817] Lustre: 57379:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1594.686588] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1594.691959] Lustre: Skipped 57 previous similar messages [ 1595.635271] Lustre: 57576:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1595.643107] Lustre: 57576:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1624.554253] Lustre: 59145:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1624.561934] Lustre: 59145:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1624.576191] LustreError: 59145:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1628.770877] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 13:42:27 (1780854147) [ 1641.958232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1641.972458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1641.981682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1641.989575] Lustre: Skipped 2 previous similar messages [ 1647.220108] Lustre: server umount lustre-MDT0000 complete [ 1647.588726] LustreError: 48136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1647.601496] LustreError: 48136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1650.803027] LustreError: 48115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780854170 with bad export cookie 10245820096745342492 [ 1650.808853] LustreError: 48115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1650.808949] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1651.154995] Lustre: server umount lustre-MDT0001 complete [ 1664.245110] Lustre: server umount lustre-OST0000 complete [ 1690.374875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.613706] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1690.842146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.501332] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1702.369639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1702.501878] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1702.552681] Lustre: 61680:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1707.222031] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1712.116581] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1715.385747] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1720.961261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1721.083773] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1726.729530] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1732.793779] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1734.754363] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1734.780188] Lustre: Skipped 18 previous similar messages [ 1734.863129] Lustre: 63907:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1746.923916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1746.925780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.960429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1746.971875] Lustre: Skipped 3 previous similar messages [ 1747.009156] Lustre: Skipped 4 previous similar messages [ 1750.257983] Lustre: server umount lustre-OST0000 complete [ 1757.154824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.172622] Lustre: Skipped 1 previous similar message [ 1760.990977] Lustre: server umount lustre-MDT0000 complete [ 1762.276507] LustreError: 61692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.296957] LustreError: 61692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1764.888958] LustreError: 61672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780854284 with bad export cookie 10245820096745345530 [ 1764.894736] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1764.908410] LustreError: 61672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1765.272380] Lustre: server umount lustre-MDT0001 complete [ 1776.372729] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 1779.177699] Key type lgssc unregistered [ 1779.508164] LNet: 65104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.515831] LNetError: 65104:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.526364] LNet: Removed LNI 192.168.204.151@tcp [ 1780.593677] Key type .llcrypt unregistered [ 1780.598996] Key type ._llcrypt unregistered [ 1808.034270] Key type ._llcrypt registered [ 1808.044418] Key type .llcrypt registered [ 1808.276667] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 1809.116534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1809.126809] alg: No test for adler32 (adler32-zlib) [ 1810.136695] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 1810.362372] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 1812.015173] Key type lgssc registered [ 1813.412197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1824.029733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1824.058728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1825.658181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.808956] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1836.000814] LustreError: 66503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1836.015045] LustreError: 66503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1841.020204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.127358] LustreError: 66504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1841.871752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1848.210803] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1854.916628] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1859.190785] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1866.396059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1866.726495] LustreError: 68103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1867.118301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.008604] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1880.395398] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1888.992805] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1902.048396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.056836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1902.070622] Lustre: Skipped 2 previous similar messages [ 1902.080029] Lustre: Skipped 2 previous similar messages [ 1907.068788] Lustre: server umount lustre-MDT0000 complete [ 1907.171412] LustreError: 66500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1907.201646] LustreError: 66500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1911.036644] LustreError: 66484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780854431 with bad export cookie 16706936686965495140 [ 1911.059403] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1911.063535] LustreError: 66484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1911.371936] Lustre: server umount lustre-MDT0001 complete [ 1925.769966] Lustre: server umount lustre-OST0000 complete [ 1952.457425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1952.766594] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1952.798849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1953.091446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.687420] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1967.826552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.940507] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1967.978868] Lustre: 71039:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1968.479716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1973.222840] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 1979.761342] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1984.180792] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1992.115174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1992.282435] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1992.688465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1999.616309] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2005.911440] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2007.888820] Lustre: 73275:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2009.774653] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2016.236358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2016.244553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.258702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2016.738143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2021.053878] Lustre: server umount lustre-OST0000 complete [ 2024.930400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2024.943449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2024.957282] Lustre: Skipped 1 previous similar message [ 2024.965340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2024.971914] Lustre: Skipped 1 previous similar message [ 2026.983706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.984096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2030.965966] Lustre: server umount lustre-MDT0000 complete [ 2032.101698] LustreError: 71051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2032.120848] LustreError: 71051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2035.758853] LustreError: 71031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780854555 with bad export cookie 16706936686965495994 [ 2035.764104] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2035.779888] LustreError: 71031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2036.392199] Lustre: server umount lustre-MDT0001 complete [ 2049.772928] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 2053.358566] Key type lgssc unregistered [ 2053.804863] LNet: 74472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.818611] LNetError: 74472:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.862809] LNet: Removed LNI 192.168.204.151@tcp [ 2054.818574] Key type .llcrypt unregistered [ 2054.822410] Key type ._llcrypt unregistered [ 2079.441867] Key type ._llcrypt registered [ 2079.446220] Key type .llcrypt registered [ 2079.549839] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2080.620768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2080.649989] alg: No test for adler32 (adler32-zlib) [ 2081.714291] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 2081.917955] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 2083.591277] Key type lgssc registered [ 2084.744122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2092.478343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2092.500917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2094.009202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.988685] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2103.273771] LustreError: 75870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.287049] LustreError: 75870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2106.547553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.302781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2111.347937] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2116.694525] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2120.065522] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2127.452364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2128.061558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2134.009316] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2139.880653] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2145.648721] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2146.379665] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2148.046432] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2148.052284] Lustre: Skipped 1 previous similar message [ 2150.892748] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2150.895995] Lustre: Skipped 2 previous similar messages [ 2156.009632] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2156.023289] Lustre: Skipped 4 previous similar messages [ 2164.834935] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2164.841537] Lustre: Skipped 9 previous similar messages [ 2181.357490] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2181.365680] Lustre: Skipped 17 previous similar messages [ 2214.076525] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2214.082503] Lustre: Skipped 37 previous similar messages [ 2242.906791] Lustre: 83633:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2246.161132] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:52:44 (1780854764) [ 2248.147271] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2250.500768] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:52:48 (1780854768) [ 2255.328584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2255.338380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.356236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2256.359353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2256.360720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2256.360915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2261.478840] Lustre: server umount lustre-OST0000 complete [ 2266.085270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2266.091810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.121287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2266.126246] Lustre: Skipped 1 previous similar message [ 2271.714188] LustreError: 75870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.824401] Lustre: server umount lustre-MDT0000 complete [ 2275.781080] LustreError: 75851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780854795 with bad export cookie 14293721210285909802 [ 2275.790955] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2275.792988] LustreError: 75851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2276.156645] Lustre: server umount lustre-MDT0001 complete [ 2286.670541] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 2289.657507] Key type lgssc unregistered [ 2290.002735] LNet: 84865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2290.012442] LNetError: 84865:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2291.049949] LNet: Removed LNI 192.168.204.151@tcp [ 2292.028975] Key type .llcrypt unregistered [ 2292.035386] Key type ._llcrypt unregistered [ 2296.739664] Key type ._llcrypt registered [ 2296.746555] Key type .llcrypt registered [ 2296.870191] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2301.304190] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules [ 2302.265918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2302.325978] alg: No test for adler32 (adler32-zlib) [ 2303.468900] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 2303.712740] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 2305.444309] Key type lgssc registered [ 2306.902240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2308.339775] LustreError: 86032:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2329.865554] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2331.237836] LustreError: 86032:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2339.625516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2339.666092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.248634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2346.245368] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2351.586497] LustreError: 87041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.601099] LustreError: 87041:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2356.455266] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:54:34 (1780854874) [ 2356.703936] LustreError: 87040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.353428] LustreError: 87072:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2361.367515] LustreError: 87072:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2361.377000] LustreError: 87072:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2361.603720] Lustre: server umount lustre-MDT0000 complete [ 2378.597375] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 2382.022269] Key type lgssc unregistered [ 2382.401892] LNet: 88532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2382.409922] LNetError: 88532:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2382.432188] LNet: Removed LNI 192.168.204.151@tcp [ 2383.223731] Key type .llcrypt unregistered [ 2383.226310] Key type ._llcrypt unregistered [ 2405.112934] Key type ._llcrypt registered [ 2405.114633] Key type .llcrypt registered [ 2405.276678] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2406.796888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2406.887119] alg: No test for adler32 (adler32-zlib) [ 2408.091342] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 2408.373251] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 2410.113111] Key type lgssc registered [ 2411.421589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2420.797695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2420.821435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.528250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2428.283435] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2432.997069] LustreError: 89737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.010533] LustreError: 89737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2438.119931] LustreError: 89738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.293202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.834260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2443.383918] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2449.563643] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2453.788957] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2461.000073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2461.735673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2468.492499] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2474.624525] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2526.646981] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:57:25 (1780855045) [ 2528.565803] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2530.942864] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:57:29 (1780855049) [ 2535.400630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.404240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2535.419228] Lustre: Skipped 2 previous similar messages [ 2535.438131] Lustre: Skipped 2 previous similar messages [ 2540.518722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2541.495934] Lustre: server umount lustre-MDT0000 complete [ 2545.633906] LustreError: 89732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.651704] LustreError: 89732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2546.134744] LustreError: 89719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780855066 with bad export cookie 4952390259815178248 [ 2546.144528] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2546.147925] LustreError: 89719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2546.724132] Lustre: server umount lustre-MDT0001 complete [ 2561.701426] Lustre: server umount lustre-OST0000 complete [ 2574.530437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2574.915779] LustreError: 93693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.081584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.201366] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2580.448971] LustreError: 93694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.466279] LustreError: 93694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2587.456194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.193083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2592.595736] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2598.981156] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2602.778202] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2616.650814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2616.858738] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2616.866337] LustreError: 93681:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2616.903980] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2616.916339] LustreError: 95551:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2616.922069] LustreError: 95551:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2616.932503] LustreError: 95551:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2617.009653] Lustre: server umount lustre-OST0000 complete [ 2617.012225] LustreError: 95551:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2626.496763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2626.745357] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2626.755826] LustreError: 93681:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2626.796537] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2626.812447] LustreError: 96017:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2626.819675] LustreError: 96017:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2626.825501] LustreError: 96017:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2626.936992] Lustre: server umount lustre-OST0000 complete [ 2626.941310] LustreError: 96017:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2637.037509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2637.351457] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2637.361802] Lustre: Found index 0 for lustre-OST0000, updating log [ 2637.370530] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2637.470295] Lustre: lustre-OST0000: new disk, initializing [ 2637.472579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2637.492579] Lustre: 96550:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2637.729175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2643.535801] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2647.505807] Lustre: Failing over lustre-OST0000 [ 2647.774419] Lustre: server umount lustre-OST0000 complete [ 2654.689569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2654.701059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.736303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2654.746648] Lustre: Skipped 2 previous similar messages [ 2657.248533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.257335] Lustre: Skipped 3 previous similar messages [ 2661.446621] Lustre: server umount lustre-MDT0000 complete [ 2662.369691] LustreError: 94407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.385804] LustreError: 94407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2665.428226] LustreError: 93675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780855185 with bad export cookie 4952390259815180985 [ 2665.432800] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2665.448560] LustreError: 93675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2665.908545] Lustre: server umount lustre-MDT0001 complete [ 2676.058712] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:59:54 (1780855194) [ 2694.442833] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2705.436273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.816364] LustreError: 98862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.008171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.632599] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2719.893984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.224723] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2725.137761] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2728.028716] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2735.566528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2736.250386] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2743.309766] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2745.460415] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2745.470392] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2745.484813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2750.638955] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:04 [ 2750.655144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2750.676649] Lustre: lustre-OST0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 2750.676784] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2750.685960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2751.972119] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2752.121624] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2752.224418] Lustre: lustre-OST0001: new disk, initializing [ 2752.229408] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2752.235890] Lustre: 101143:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2752.578806] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2752.585411] Lustre: Skipped 2 previous similar messages [ 2758.625112] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2760.322290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2760.332231] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2760.438485] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2766.174240] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2770.602308] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2776.036902] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.049508] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2776.062169] Lustre: Skipped 3 previous similar messages [ 2776.082290] Lustre: Skipped 2 previous similar messages [ 2780.625747] Lustre: server umount lustre-MDT0001 complete [ 2781.160958] LustreError: 98862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.195859] LustreError: 98862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2790.831535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2790.923985] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2790.927475] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2790.974982] Lustre: 98850:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2792.056624] LustreError: 102663:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2792.064900] LustreError: 102663:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2792.077243] LustreError: 98854:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2792.182125] LustreError: 102671:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.151@tcp: cfg command failed: rc = -17 [ 2792.190040] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2792.190040] [ 2795.867079] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 2797.050090] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2806.004501] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 14:02:04 (1780855324) [ 2824.890473] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 2842.896356] Lustre: 104710:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2866.955129] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2870.805787] Lustre: 105845:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2965.473824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2965.488425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.507099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2965.512304] Lustre: Skipped 1 previous similar message [ 2970.972743] Lustre: server umount lustre-MDT0000 complete [ 2971.108334] LustreError: 98858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.133835] LustreError: 98858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2974.785842] LustreError: 98842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780855494 with bad export cookie 4952390259815181993 [ 2974.792683] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2974.804528] LustreError: 98842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2975.274337] Lustre: server umount lustre-MDT0001 complete [ 2989.789566] Lustre: server umount lustre-OST0000 complete [ 2992.095143] Lustre: 89050:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780855496/real 1780855496] req@ffff9bc9ce6d2680 x1867361737336192/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780855512 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2992.131841] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.140495] Lustre: Skipped 3 previous similar messages [ 2993.922841] Lustre: server umount lustre-OST0001 complete [ 2999.755027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3009.921756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3018.091049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3027.556510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3044.496488] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 3053.219394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.240166] Lustre: lustre-MDT0000: reset Object Index mappings [ 3053.716671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3053.720845] Lustre: Skipped 1 previous similar message [ 3057.430657] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3064.951139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3064.972374] Lustre: lustre-MDT0001: reset Object Index mappings [ 3069.469544] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3072.133772] Lustre: 110896:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3077.883181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3083.763546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3084.276715] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3090.933954] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3092.457635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3098.105391] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3098.105585] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3098.393241] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3104.895684] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3107.838459] Lustre: 112739:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3156.329170] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:07:55 (1780855675) [ 3185.131746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3185.136528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.139810] Lustre: Skipped 3 previous similar messages [ 3185.147383] Lustre: Skipped 6 previous similar messages [ 3189.804677] Lustre: server umount lustre-MDT0000 complete [ 3190.243834] LustreError: 110502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.257399] LustreError: 110502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 3193.470331] LustreError: 109765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780855713 with bad export cookie 4952390259815229047 [ 3193.483444] LustreError: 109765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3193.491747] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3199.962312] Lustre: server umount lustre-MDT0001 complete [ 3209.365902] Lustre: server umount lustre-OST0000 complete [ 3227.103118] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3227.225462] Lustre: server umount lustre-OST0001 complete [ 3232.546450] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 3241.607457] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 3277.026823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3277.282258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3277.312483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3277.404985] Lustre: lustre-MDT0000: new disk, initializing [ 3277.496649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.504418] Lustre: Skipped 3 previous similar messages [ 3277.516924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3281.370744] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3287.573183] Lustre: server umount lustre-MDT0000 complete [ 3296.151464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.374359] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3296.395319] Lustre: 116955:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3296.411720] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3300.557479] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3304.942091] Lustre: server umount lustre-MDT0000 complete [ 3311.325961] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 14:10:30 (1780855830) [ 3312.617023] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3313.911730] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 14:10:32 (1780855832) [ 3326.305251] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 3333.270981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.064174] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3344.498499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3344.618933] Lustre: 118970:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3344.667268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3344.673431] Lustre: Skipped 1 previous similar message [ 3344.766413] Lustre: lustre-MDT0001: new disk, initializing [ 3344.896963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3344.905333] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3348.579480] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3352.727775] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3358.071338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3358.273605] Lustre: lustre-OST0000: new disk, initializing [ 3358.277787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3358.287387] Lustre: 120513:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3360.293378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3360.306138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3360.417475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3363.712545] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3372.228292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.367513] Lustre: lustre-OST0001: new disk, initializing [ 3372.370088] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3372.378316] Lustre: 121368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3374.143528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3374.151664] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3374.247989] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3377.247838] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 3384.676921] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3387.355878] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3704.099489] Lustre: 127754:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3704.109413] Lustre: 127754:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3704.120899] Lustre: 127754:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3704.126632] Lustre: 127754:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3704.133984] Lustre: 127754:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3704.140993] Lustre: 127754:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3931.601431] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:20:50 (1780856450) [ 3949.024304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3949.029323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.036379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3949.037395] Lustre: Skipped 5 previous similar messages [ 3949.046420] Lustre: Skipped 7 previous similar messages [ 3951.001447] Lustre: server umount lustre-MDT0000 complete [ 3953.075474] LustreError: 127767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780856473 with bad export cookie 4952390259815297353 [ 3953.078218] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3953.081986] LustreError: 127767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3954.143998] LustreError: 118978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3954.145603] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3954.156909] LustreError: 118978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3954.178520] Lustre: Skipped 1 previous similar message [ 3958.183416] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3958.186522] Lustre: Skipped 2 previous similar messages [ 3959.388921] Lustre: server umount lustre-MDT0001 complete [ 3976.159138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3976.221104] Lustre: server umount lustre-OST0000 complete [ 3984.631814] Lustre: server umount lustre-OST0001 complete [ 3988.984239] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 3996.817633] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 4019.759164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4019.914607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4019.939183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4019.982504] Lustre: lustre-MDT0000: new disk, initializing [ 4020.037658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4020.046459] Lustre: Skipped 5 previous similar messages [ 4020.058780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4022.385840] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 4028.973133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4029.048091] Lustre: 130769:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4029.099374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4029.102274] Lustre: Skipped 1 previous similar message [ 4029.148438] Lustre: lustre-MDT0001: new disk, initializing [ 4029.231656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4029.235615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4031.484523] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 4035.904809] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4037.874340] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4041.548665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4041.682812] Lustre: lustre-OST0000: new disk, initializing [ 4041.686752] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4041.691371] Lustre: 132512:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4042.903555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4042.907633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4042.965230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4044.916676] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 4049.520556] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4055.115840] Lustre: lustre-MDD0000: changelog on [ 4056.679271] Lustre: lustre-MDD0001: changelog on [ 4940.416113] Lustre: lustre-MDD0001: changelog off [ 4941.130492] Lustre: 130776:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4941.533503] Lustre: lustre-MDD0000: changelog off [ 4944.358298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4944.359057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4944.364882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4944.370371] Lustre: Skipped 2 previous similar messages [ 4949.405919] Lustre: server umount lustre-OST0000 complete [ 4951.008785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4951.013814] LustreError: Skipped 1 previous similar message [ 4951.019413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4951.027677] Lustre: Skipped 1 previous similar message [ 4951.030660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4954.593106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4959.713181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4959.719257] Lustre: Skipped 3 previous similar messages [ 4965.343143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4965.476544] Lustre: server umount lustre-MDT0000 complete [ 4967.211542] LustreError: 130760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857487 with bad export cookie 4952390259816110067 [ 4967.215079] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4967.216606] LustreError: 130760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4967.389780] Lustre: server umount lustre-MDT0001 complete [ 4971.572098] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 4972.719633] Key type lgssc unregistered [ 4972.857716] LNet: 141292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4972.861067] LNetError: 141292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4972.869814] LNet: Removed LNI 192.168.204.151@tcp [ 4973.243152] Key type .llcrypt unregistered [ 4973.245402] Key type ._llcrypt unregistered [ 4981.815669] Key type ._llcrypt registered [ 4981.816956] Key type .llcrypt registered [ 4981.861970] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 4989.541730] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 4990.025150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4990.034553] alg: No test for adler32 (adler32-zlib) [ 4990.920904] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 4991.035039] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 4992.655525] Key type lgssc registered [ 4993.177792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5008.980225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5008.987650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5010.108890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5010.126278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5010.179323] Lustre: lustre-MDT0000: new disk, initializing [ 5010.231209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5010.245656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5012.102611] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5017.684841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5017.741581] Lustre: 144434:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5017.773271] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5017.775574] Lustre: Skipped 1 previous similar message [ 5017.823901] Lustre: lustre-MDT0001: new disk, initializing [ 5017.874848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5017.895304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5017.902255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5019.790545] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5023.629399] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5025.113423] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5027.986685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5028.110329] Lustre: lustre-OST0000: new disk, initializing [ 5028.112631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5028.116829] Lustre: 146176:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5028.160647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5030.664146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5030.668613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5030.717118] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5030.726714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5034.644970] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5037.339981] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_module obdecho/obdecho [ 5038.173590] LustreError: 147049:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5038.177693] LustreError: 147049:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5038.964296] LustreError: 147145:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 5043.682395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5043.687718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5043.702907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5044.194442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5044.201040] Lustre: Skipped 1 previous similar message [ 5044.203884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5044.205967] Lustre: Skipped 2 previous similar messages [ 5049.314190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5049.317763] Lustre: Skipped 2 previous similar messages [ 5049.660748] Lustre: server umount lustre-MDT0000 complete [ 5051.274284] LustreError: 144427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857571 with bad export cookie 16442156447234804150 [ 5051.277582] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5051.279037] LustreError: 144427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5051.402414] Lustre: server umount lustre-MDT0001 complete [ 5063.402932] Lustre: server umount lustre-OST0000 complete [ 5070.895298] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 5074.887641] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 5088.551993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5088.657486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5088.671742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5088.709487] Lustre: lustre-MDT0000: new disk, initializing [ 5088.745839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5088.753561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5090.214692] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5094.695358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5094.748653] Lustre: 150819:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5094.775509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5094.777572] Lustre: Skipped 1 previous similar message [ 5094.813337] Lustre: lustre-MDT0001: new disk, initializing [ 5094.846406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5094.859814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5094.863239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5096.231607] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5099.353438] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5100.584825] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5103.005101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5103.102610] Lustre: lustre-OST0000: new disk, initializing [ 5103.104217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5103.106846] Lustre: 152561:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5103.135531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5104.953490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5104.956481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5104.987967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5105.034907] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5108.195775] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5109.522594] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5109.590796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5110.887926] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5110.966837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5113.683303] Lustre: server umount lustre-OST0000 complete [ 5115.360900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5115.361189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5115.365551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5115.370326] Lustre: Skipped 1 previous similar message [ 5120.480453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5120.482993] Lustre: Skipped 1 previous similar message [ 5125.627973] Lustre: server umount lustre-MDT0000 complete [ 5126.928152] LustreError: 151597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857646 with bad export cookie 16442156447234805109 [ 5126.931309] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5126.933044] LustreError: 151597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5136.415635] Lustre: server umount lustre-MDT0001 complete [ 5137.919291] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:40:57 (1780857657) [ 5138.608392] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5139.388897] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:40:58 (1780857658) [ 5142.410330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5142.598195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5144.018028] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5146.956299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5148.489291] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5150.401210] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5151.698552] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5154.045421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5156.216615] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5158.075076] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5172.708465] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5172.711199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5172.711289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5172.714225] Lustre: Skipped 3 previous similar messages [ 5172.722760] Lustre: Skipped 4 previous similar messages [ 5176.805335] Lustre: server umount lustre-MDT0000 complete [ 5177.824385] LustreError: 154827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5177.830265] LustreError: 154827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5178.141186] LustreError: 156441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857698 with bad export cookie 16442156447234805970 [ 5178.144283] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5178.146377] LustreError: 156441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5178.643389] LustreError: 156890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5182.944194] LustreError: 154822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5182.944448] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5184.322862] Lustre: server umount lustre-MDT0001 complete [ 5186.749552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5191.554806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5191.656279] LustreError: 158345:0:(llog.c:1646:llog_backup()) MGC192.168.204.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5191.659754] Lustre: 158345:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5208.544314] LustreError: 158350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5208.551668] LustreError: 158350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5213.663834] LustreError: 158368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5213.668527] LustreError: 158368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5223.903965] LustreError: 158350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5223.910431] LustreError: 158350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5242.849284] LustreError: 158352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5242.855493] LustreError: 158352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 5252.703357] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5252.706052] Lustre: 158345:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5274.899592] LustreError: 158351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5274.907732] LustreError: 158351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [ 5293.541801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5293.544631] Lustre: Skipped 2 previous similar messages [ 5294.862977] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5297.891358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5298.658974] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5299.681332] LustreError: 142659:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005bc08c1f x1867364445554944/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5301.203902] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5303.103443] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5305.317855] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5305.690781] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5306.943706] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5310.620768] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5316.049521] Lustre: server umount lustre-MDT0000 complete [ 5317.345257] LustreError: 154808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857837 with bad export cookie 16442156447234812417 [ 5317.348565] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5317.349797] LustreError: 154808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5320.672263] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5320.673331] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5320.679327] Lustre: Skipped 1 previous similar message [ 5320.682843] Lustre: Skipped 5 previous similar messages [ 5323.511146] Lustre: server umount lustre-MDT0001 complete [ 5325.164360] Lustre: server umount lustre-OST0000 complete [ 5333.694131] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 5337.800826] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 5352.531169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5352.625576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5352.641955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5352.681391] Lustre: lustre-MDT0000: new disk, initializing [ 5352.715902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5354.117151] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5358.618797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5358.666169] Lustre: 163154:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5358.764948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5358.767715] Lustre: Skipped 2 previous similar messages [ 5358.783795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5360.266540] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5363.630098] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5364.992061] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5367.410976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5367.513586] Lustre: 164897:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5369.148828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5369.192309] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5369.573648] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5372.853182] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5374.407258] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5374.490278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5375.999427] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5376.090368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5379.555108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5379.557286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5379.557877] LustreError: Skipped 1 previous similar message [ 5379.563418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5379.567411] Lustre: Skipped 2 previous similar messages [ 5391.327133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5391.395586] Lustre: server umount lustre-OST0000 complete [ 5394.915368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5407.199148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5407.267239] Lustre: server umount lustre-MDT0000 complete [ 5408.614244] LustreError: 163191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857928 with bad export cookie 16442156447234829224 [ 5408.617663] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5408.618826] LustreError: 163191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5408.731589] Lustre: server umount lustre-MDT0001 complete [ 5412.017938] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:45:31 (1780857931) [ 5414.375203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5414.554121] LustreError: 167158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5414.560979] LustreError: 167158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 5416.011933] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5419.055414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5420.473769] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5422.461558] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5423.830416] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5426.244308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5428.266828] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5430.186756] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5444.529166] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:46:04 (1780857964) [ 5450.208771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5450.209905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.210934] LustreError: Skipped 1 previous similar message [ 5450.215115] Lustre: Skipped 3 previous similar messages [ 5450.215784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5450.221711] Lustre: Skipped 11 previous similar messages [ 5451.597090] Lustre: server umount lustre-OST0000 complete [ 5467.103098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5467.156915] Lustre: server umount lustre-MDT0000 complete [ 5468.375251] LustreError: 167140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857988 with bad export cookie 16442156447234830092 [ 5468.378465] LustreError: 167140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5468.378613] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5468.485067] Lustre: server umount lustre-MDT0001 complete [ 5472.115884] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 5473.072467] Key type lgssc unregistered [ 5473.204311] LNet: 170550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5473.206730] LNetError: 170550:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5473.216337] LNet: Removed LNI 192.168.204.151@tcp [ 5473.517931] Key type .llcrypt unregistered [ 5473.519132] Key type ._llcrypt unregistered [ 5480.739092] Key type ._llcrypt registered [ 5480.740778] Key type .llcrypt registered [ 5480.778252] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 5481.073945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5481.138852] alg: No test for adler32 (adler32-zlib) [ 5481.987126] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 5482.064702] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 5483.639137] Key type lgssc registered [ 5483.991724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5486.706130] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5486.714035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5502.175402] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5507.295199] LustreError: 171753:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.151@tcp: failed processing log, type 4: rc = -110 [ 5512.415518] LustreError: 171753:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5512.419863] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5512.421748] LustreError: MGC192.168.204.151@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5512.425499] LustreError: 171753:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5512.428283] LustreError: 171753:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 5512.430807] LustreError: 171753:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 5512.454626] Lustre: server umount lustre-OST0000 complete [ 5512.456537] LustreError: 171753:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5515.298640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5516.452624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5517.755539] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5520.574730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5520.692581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5521.976983] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5523.709204] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5524.867396] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5527.096175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5527.161680] LustreError: 173839:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5527.166736] Lustre: 173839:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.151@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5527.211089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.101814] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5530.813646] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5535.712716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5535.714922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5535.716897] Lustre: Skipped 2 previous similar messages [ 5535.720797] Lustre: Skipped 2 previous similar messages [ 5537.617271] Lustre: server umount lustre-MDT0000 complete [ 5538.839453] LustreError: 172969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858058 with bad export cookie 8595780752998020899 [ 5538.842929] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5538.843704] LustreError: 172969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5538.950518] Lustre: server umount lustre-MDT0001 complete [ 5550.557447] Lustre: server umount lustre-OST0000 complete [ 5553.556706] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 5556.980575] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 5568.817092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5568.908148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5568.921706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5568.957585] Lustre: lustre-MDT0000: new disk, initializing [ 5568.986779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5568.995212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5570.349522] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5574.572454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5574.613035] Lustre: 177209:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5574.636134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5574.637946] Lustre: Skipped 1 previous similar message [ 5574.672576] Lustre: lustre-MDT0001: new disk, initializing [ 5574.705616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5574.717600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5574.720762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5576.079205] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5579.238139] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5580.531490] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5582.855155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5582.965308] Lustre: lustre-OST0000: new disk, initializing [ 5582.968112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5582.971900] Lustre: 178953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5583.004726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.637682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5584.641750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5584.687714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5585.060655] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5588.398801] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5589.871568] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5589.959450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5591.421405] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5591.500124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5595.108717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5595.109348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5595.112322] LustreError: Skipped 1 previous similar message [ 5595.118086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5595.122749] Lustre: Skipped 1 previous similar message [ 5598.297078] Lustre: server umount lustre-OST0000 complete [ 5600.226462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5600.226505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5600.230075] Lustre: Skipped 1 previous similar message [ 5600.230663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5600.237402] Lustre: Skipped 1 previous similar message [ 5605.344035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5605.346581] Lustre: Skipped 1 previous similar message [ 5605.784513] Lustre: server umount lustre-MDT0000 complete [ 5607.113207] LustreError: 177203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858127 with bad export cookie 8595780752998021585 [ 5607.115507] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5607.117525] LustreError: 177203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5607.244459] Lustre: server umount lustre-MDT0001 complete [ 5610.474117] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:48:50 (1780858130) [ 5616.563535] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 5617.550965] Key type lgssc unregistered [ 5617.669329] LNet: 181731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5617.672577] LNetError: 181731:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5617.681287] LNet: Removed LNI 192.168.204.151@tcp [ 5617.966493] Key type .llcrypt unregistered [ 5617.967376] Key type ._llcrypt unregistered [ 5625.014730] Key type ._llcrypt registered [ 5625.015831] Key type .llcrypt registered [ 5625.054205] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 5625.339023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5625.351769] alg: No test for adler32 (adler32-zlib) [ 5626.199525] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 5626.280200] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 5627.855156] Key type lgssc registered [ 5628.222588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5630.954134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5630.961443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5632.143308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5633.523618] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5636.533356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.660816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5637.983820] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5639.926867] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5641.215957] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5643.805042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5643.878648] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5643.880092] LustreError: 184544:0:(llog.c:1687:llog_backup()) MGC192.168.204.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5643.883693] Lustre: 184544:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.151@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5643.920423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5645.906855] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5647.797398] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5651.935838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5651.938325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5651.942350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5652.448394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5652.449695] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5654.675532] Lustre: server umount lustre-MDT0000 complete [ 5655.921616] LustreError: 182890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858175 with bad export cookie 7518348370470153529 [ 5655.922769] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5655.926641] LustreError: 182890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5656.035283] Lustre: server umount lustre-MDT0001 complete [ 5667.606222] Lustre: server umount lustre-OST0000 complete [ 5670.784609] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 5674.497817] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 5687.171200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5687.264748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5687.277097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5687.312383] Lustre: lustre-MDT0000: new disk, initializing [ 5687.336561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5687.344074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5688.702663] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5692.991845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5693.038272] Lustre: 187915:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5693.061434] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5693.063331] Lustre: Skipped 1 previous similar message [ 5693.065361] LustreError: 188680:0:(llog.c:1687:llog_backup()) MGC192.168.204.151@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5693.069605] LustreError: 188680:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 5693.072941] Lustre: 188680:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.151@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5693.083239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5693.086134] Lustre: Skipped 1 previous similar message [ 5693.126155] Lustre: lustre-MDT0001: new disk, initializing [ 5693.161544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5693.174980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5693.179449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5694.511799] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5697.607830] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5698.878789] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5701.211482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5701.306089] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5701.308064] Lustre: Skipped 1 previous similar message [ 5701.309508] LustreError: 189631:0:(llog.c:1687:llog_backup()) MGC192.168.204.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5701.312899] LustreError: 189631:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 5701.314897] Lustre: 189631:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.151@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5701.317583] Lustre: 189631:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5701.326814] Lustre: lustre-OST0000: new disk, initializing [ 5701.329434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5701.333264] Lustre: 189655:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5701.365894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5703.253410] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5706.372716] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5707.769788] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5708.279565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5708.283040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5708.315136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5708.858495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5710.181603] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5710.254963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5713.378083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5713.378476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.379642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5713.379649] Lustre: Skipped 2 previous similar messages [ 5713.380940] LustreError: Skipped 1 previous similar message [ 5713.393425] Lustre: Skipped 2 previous similar messages [ 5717.005313] Lustre: server umount lustre-OST0000 complete [ 5718.496270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.496692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5718.501620] Lustre: Skipped 1 previous similar message [ 5718.503344] Lustre: Skipped 1 previous similar message [ 5723.616139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5723.618812] Lustre: Skipped 2 previous similar messages [ 5724.512766] Lustre: server umount lustre-MDT0000 complete [ 5725.789208] LustreError: 187907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858245 with bad export cookie 7518348370470154229 [ 5725.792663] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5725.794090] LustreError: 187907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5725.902421] Lustre: server umount lustre-MDT0001 complete [ 5728.970233] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:50:48 (1780858248) [ 5734.493439] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 5738.147569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5738.343410] LustreError: 192619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5738.350746] LustreError: 192619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5738.361112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5739.737626] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5742.799605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5744.319287] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5745.364839] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5747.896906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5750.202665] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5753.409049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5753.478957] Lustre: lustre-OST0001: new disk, initializing [ 5753.482206] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5753.486210] Lustre: 194871:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5755.722030] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5758.984650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5758.989034] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5759.024125] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5761.592708] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5765.474256] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5766.244242] Lustre: DEBUG MARKER: ADD OST3 [ 5768.157223] loop: module loaded [ 5768.176936] loop0: detected capacity change from 0 to 204800000 [ 5768.315644] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5768.321552] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5768.327686] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5770.728951] loop0: detected capacity change from 0 to 204800000 [ 5772.874739] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5772.946272] Lustre: lustre-OST0003: new disk, initializing [ 5772.949115] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5772.952772] Lustre: 196589:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5772.995525] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5772.998672] Lustre: Skipped 3 previous similar messages [ 5773.460071] Lustre: DEBUG MARKER: STOP OST3 [ 5774.208551] Lustre: Failing over lustre-OST0003 [ 5774.250386] LustreError: 194880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.262636] Lustre: server umount lustre-OST0003 complete [ 5775.408944] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5779.211189] Lustre: DEBUG MARKER: START OST3 again [ 5779.732917] LustreError: 194067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5779.740649] LustreError: 194067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5781.744536] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5783.939834] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5787.115913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5787.120607] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5787.132507] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5797.519067] Lustre: Failing over lustre-OST0003 [ 5797.557690] Lustre: server umount lustre-OST0003 complete [ 5802.465539] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5802.468440] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5802.471500] LustreError: 194066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5802.472442] Lustre: Skipped 1 previous similar message [ 5802.476844] LustreError: 194066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5807.592078] LustreError: 194067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5807.596310] LustreError: 194067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5812.707636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5812.710812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5812.713038] Lustre: Skipped 3 previous similar messages [ 5812.714711] Lustre: Skipped 2 previous similar messages [ 5816.152582] Lustre: server umount lustre-MDT0000 complete [ 5817.367514] LustreError: 192600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858337 with bad export cookie 7518348370470155097 [ 5817.372063] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5817.373176] LustreError: 192600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5817.824107] LustreError: 192615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5817.830468] LustreError: 192615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5823.605929] Lustre: server umount lustre-MDT0001 complete [ 5824.978254] Lustre: server umount lustre-OST0000 complete [ 5826.342431] Lustre: server umount lustre-OST0001 complete [ 5828.497749] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 5831.754207] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 5845.290662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.387269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5845.401079] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5845.443456] Lustre: lustre-MDT0000: new disk, initializing [ 5845.477336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5845.480138] Lustre: Skipped 1 previous similar message [ 5845.487544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5846.917602] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5851.299574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.348535] Lustre: 201930:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5851.467361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5852.884526] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5856.021225] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5857.215516] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5859.421762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5859.523313] Lustre: 203674:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5860.603547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5860.646739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5861.548211] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5864.787639] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5866.319652] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5866.402819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5867.885482] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5867.967930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5870.740617] Lustre: server umount lustre-OST0000 complete [ 5872.095836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5872.098151] LustreError: Skipped 1 previous similar message [ 5872.099699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5872.103937] Lustre: Skipped 2 previous similar messages [ 5872.105955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5872.108513] Lustre: Skipped 5 previous similar messages [ 5882.365195] Lustre: server umount lustre-MDT0000 complete [ 5883.714203] LustreError: 203673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858403 with bad export cookie 7518348370470157358 [ 5883.718094] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5883.718696] LustreError: 203673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5893.140576] Lustre: server umount lustre-MDT0001 complete [ 5894.397858] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:53:33 (1780858413) [ 5900.130817] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 5903.964130] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 5916.221369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5916.311360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5916.323896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5916.325573] Lustre: Skipped 3 previous similar messages [ 5916.361684] Lustre: lustre-MDT0000: new disk, initializing [ 5916.364179] Lustre: Skipped 2 previous similar messages [ 5916.393805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5916.396077] Lustre: Skipped 2 previous similar messages [ 5917.660370] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5921.820493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5921.861335] Lustre: 208098:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5921.959959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5923.238448] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5926.230798] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5927.451265] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5929.660742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5929.764827] Lustre: 209840:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5931.099188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5931.687334] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5934.766669] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5936.153098] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5936.224421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5937.552399] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5937.621119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5941.221532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5941.221544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5941.225245] Lustre: Skipped 1 previous similar message [ 5941.229916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5941.233115] Lustre: Skipped 5 previous similar messages [ 5944.405689] Lustre: server umount lustre-OST0000 complete [ 5951.768830] Lustre: server umount lustre-MDT0000 complete [ 5952.972256] LustreError: 209839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858472 with bad export cookie 7518348370470158240 [ 5952.975656] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5952.976511] LustreError: 209839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5953.092383] Lustre: server umount lustre-MDT0001 complete [ 5956.032550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5956.188218] LustreError: 211906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.194543] LustreError: 211906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5957.479822] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5960.351698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5961.723248] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5963.464028] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5964.677087] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5966.818752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5968.777665] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 5970.454350] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5990.879813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5990.882871] LustreError: Skipped 1 previous similar message [ 5993.296170] Lustre: server umount lustre-OST0000 complete [ 6000.669662] Lustre: server umount lustre-MDT0000 complete [ 6001.869348] LustreError: 211886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858521 with bad export cookie 7518348370470159122 [ 6001.871798] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6001.872084] LustreError: 211886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6001.971774] Lustre: server umount lustre-MDT0001 complete [ 6106.085523] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 6107.147136] Key type lgssc unregistered [ 6107.287385] LNet: 215234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6107.290372] LNetError: 215234:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6107.298453] LNet: Removed LNI 192.168.204.151@tcp [ 6107.595108] Key type .llcrypt unregistered [ 6107.596153] Key type ._llcrypt unregistered [ 6115.764046] Key type ._llcrypt registered [ 6115.765267] Key type .llcrypt registered [ 6115.798364] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 6121.802428] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6122.125063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6122.160087] alg: No test for adler32 (adler32-zlib) [ 6123.007097] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 6123.084912] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 6124.663132] Key type lgssc registered [ 6124.992177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6136.155746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6136.160483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6137.248583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6137.259936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6137.292988] Lustre: lustre-MDT0000: new disk, initializing [ 6137.316266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6137.322978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6138.558258] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6142.755735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6142.803177] Lustre: 218526:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6142.827133] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6142.829462] Lustre: Skipped 1 previous similar message [ 6142.867943] Lustre: lustre-MDT0001: new disk, initializing [ 6142.906233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6142.920562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6142.925509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6144.314475] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6147.379386] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6148.655995] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6150.947878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6151.055507] Lustre: lustre-OST0000: new disk, initializing [ 6151.057370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6151.061146] Lustre: 220266:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6151.089182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6152.761736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6152.765145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6152.799954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6153.033667] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6156.147031] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6157.441918] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6157.506379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6158.754314] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6158.822777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6162.913988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.914520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6162.917852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6162.920895] Lustre: Skipped 1 previous similar message [ 6165.533662] Lustre: server umount lustre-OST0000 complete [ 6168.033173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6168.034399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6168.038740] Lustre: Skipped 2 previous similar messages [ 6173.151956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6173.154873] Lustre: Skipped 2 previous similar messages [ 6178.272345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6178.274817] Lustre: Skipped 1 previous similar message [ 6181.343278] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6181.409386] Lustre: server umount lustre-MDT0000 complete [ 6182.634074] LustreError: 218517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858702 with bad export cookie 2626127190611514664 [ 6182.636451] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6182.638879] LustreError: 218517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6182.747433] Lustre: server umount lustre-MDT0001 complete [ 6188.413973] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6191.714127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6191.855631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6193.088827] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6195.883715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6195.998325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6197.236777] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6198.141441] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6200.275129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6202.182932] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6204.960494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6205.024623] Lustre: lustre-OST0001: new disk, initializing [ 6205.026567] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6205.029658] Lustre: 225284:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6205.058043] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6205.060971] Lustre: Skipped 1 previous similar message [ 6206.848315] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6212.156391] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6213.118304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6213.121065] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6213.149594] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6215.676728] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6219.292178] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:58:58 (1780858738) [ 6221.792987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6221.796330] LustreError: Skipped 1 previous similar message [ 6221.798148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6221.802370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6226.078895] Lustre: server umount lustre-MDT0000 complete [ 6227.349184] LustreError: 226095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858747 with bad export cookie 2626127190611515532 [ 6227.352191] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6227.352714] LustreError: 226095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6227.475015] Lustre: server umount lustre-MDT0001 complete [ 6239.041554] Lustre: server umount lustre-OST0000 complete [ 6250.522811] Lustre: server umount lustre-OST0001 complete [ 6252.603123] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 6255.663556] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6267.297715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6267.394512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6267.412463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6267.448693] Lustre: lustre-MDT0000: new disk, initializing [ 6267.479112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6267.486358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6268.867315] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6272.962096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6272.999470] Lustre: 229172:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6273.097158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6274.360184] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6277.416255] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6278.595669] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6280.824185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6280.947773] Lustre: lustre-OST0000: new disk, initializing [ 6280.949650] Lustre: Skipped 1 previous similar message [ 6280.951743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6280.954382] Lustre: Skipped 2 previous similar messages [ 6280.956812] Lustre: 230918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6282.165812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6282.169934] Lustre: Skipped 1 previous similar message [ 6282.171728] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6282.195102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6282.867284] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6285.879636] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6287.233568] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6287.304964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6288.592513] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6288.657083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6292.448595] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6292.450509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6292.451163] LustreError: Skipped 1 previous similar message [ 6292.455257] Lustre: Skipped 3 previous similar messages [ 6292.457431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6292.460317] Lustre: Skipped 5 previous similar messages [ 6295.376971] Lustre: server umount lustre-OST0000 complete [ 6297.568818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.573567] Lustre: Skipped 1 previous similar message [ 6302.748669] Lustre: server umount lustre-MDT0000 complete [ 6303.987483] LustreError: 229165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858823 with bad export cookie 2626127190611517653 [ 6303.990315] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6303.990952] LustreError: 229165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6304.120623] Lustre: server umount lustre-MDT0001 complete [ 6306.956955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6307.104565] LustreError: 232983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6307.110940] LustreError: 232983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6307.128316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6307.130888] Lustre: Skipped 2 previous similar messages [ 6308.359865] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6311.217624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6312.620721] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6314.395308] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6315.556248] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6317.827295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6319.749177] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6321.448981] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6464.479776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6464.482937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6464.486823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6464.489142] Lustre: Skipped 4 previous similar messages [ 6469.715302] Lustre: server umount lustre-OST0000 complete [ 6475.231968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6475.235321] LustreError: Skipped 1 previous similar message [ 6477.146667] Lustre: server umount lustre-MDT0000 complete [ 6478.437490] LustreError: 232965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858998 with bad export cookie 2626127190611518535 [ 6478.440232] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6478.442508] LustreError: 232965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6478.560189] Lustre: server umount lustre-MDT0001 complete [ 6584.793103] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 6585.749862] Key type lgssc unregistered [ 6585.872351] LNet: 236248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6585.874988] LNetError: 236248:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6585.884344] LNet: Removed LNI 192.168.204.151@tcp [ 6586.183160] Key type .llcrypt unregistered [ 6586.184513] Key type ._llcrypt unregistered [ 6593.964103] Key type ._llcrypt registered [ 6593.965057] Key type .llcrypt registered [ 6593.999517] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 6599.988126] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6600.255489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6600.265574] alg: No test for adler32 (adler32-zlib) [ 6601.113160] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 6601.186064] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 6602.767166] Key type lgssc registered [ 6603.099262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6616.322781] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6619.457475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6619.463383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.543509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6620.554472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6620.589060] Lustre: lustre-MDT0000: new disk, initializing [ 6620.609828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6620.615866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6621.817568] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6625.930968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6625.976580] Lustre: 240235:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6626.003932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6626.006833] Lustre: Skipped 1 previous similar message [ 6626.050769] Lustre: lustre-MDT0001: new disk, initializing [ 6626.087514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6626.101235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6626.105818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6627.544304] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6629.794034] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6632.030483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6632.144869] Lustre: lustre-OST0000: new disk, initializing [ 6632.147596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6632.151089] Lustre: 241829:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6632.190099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6634.111503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6634.114862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6634.165145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6634.192114] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6638.300288] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6638.367831] Lustre: lustre-OST0001: new disk, initializing [ 6638.369267] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6638.371270] Lustre: 242682:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6638.410894] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6640.274685] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6642.686541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6642.689347] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6642.720843] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6645.571573] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6651.187549] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6656.991904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6656.993903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6656.997586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6658.019838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6658.020420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6660.069752] Lustre: server umount lustre-MDT0000 complete [ 6661.331405] LustreError: 240226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859181 with bad export cookie 8563679210986678297 [ 6661.334519] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6661.335393] LustreError: 240226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6661.467915] Lustre: server umount lustre-MDT0001 complete [ 6672.325534] Lustre: server umount lustre-OST0000 complete [ 6683.878595] Lustre: server umount lustre-OST0001 complete [ 6688.539640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6691.624758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6694.410143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6697.528159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6703.825908] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6707.421165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6707.428948] Lustre: lustre-MDT0000: reset Object Index mappings [ 6707.613662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6709.017241] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6711.985158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6711.992433] Lustre: lustre-MDT0001: reset Object Index mappings [ 6713.545380] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6714.573725] Lustre: 248754:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6716.705095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6716.817286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6716.819542] Lustre: Skipped 1 previous similar message [ 6717.859775] LustreError: 249106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6717.862678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6717.866034] LustreError: 249106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6718.598504] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6721.252460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6722.346315] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6723.071166] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6726.160837] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6727.414435] Lustre: 250588:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6732.772241] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.774366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6732.776185] Lustre: Skipped 4 previous similar messages [ 6732.776856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6732.778423] Lustre: Skipped 6 previous similar messages [ 6734.752240] Lustre: server umount lustre-MDT0000 complete [ 6736.060284] LustreError: 250590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859256 with bad export cookie 8563679210986685073 [ 6736.064025] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6736.065368] LustreError: 250590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6736.191320] Lustre: server umount lustre-MDT0001 complete [ 6747.074094] Lustre: server umount lustre-OST0000 complete [ 6758.556027] Lustre: server umount lustre-OST0001 complete [ 6765.024383] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:08:04 (1780859284) [ 6769.910806] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 6772.977362] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6784.086443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6784.174380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6784.184513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6784.218158] Lustre: lustre-MDT0000: new disk, initializing [ 6784.241570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6784.244885] Lustre: Skipped 1 previous similar message [ 6784.251334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6785.435172] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6789.244321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6789.284614] Lustre: 254800:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6789.372868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6790.553126] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6793.631624] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6794.955294] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6797.346835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6797.451077] Lustre: lustre-OST0000: new disk, initializing [ 6797.452533] Lustre: Skipped 1 previous similar message [ 6797.454066] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6797.456797] Lustre: Skipped 2 previous similar messages [ 6797.460276] Lustre: 256542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6798.585817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6798.588491] Lustre: Skipped 1 previous similar message [ 6798.590471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6798.623873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6799.414809] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6802.443530] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6803.756291] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6803.827225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6805.053132] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6805.112420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6809.058617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6809.059247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6809.060100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6809.064621] Lustre: Skipped 2 previous similar messages [ 6809.067552] LustreError: Skipped 1 previous similar message [ 6814.177759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6814.179721] Lustre: Skipped 2 previous similar messages [ 6820.319163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6820.377487] Lustre: server umount lustre-OST0000 complete [ 6824.418235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6824.418923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6824.421638] Lustre: Skipped 1 previous similar message [ 6824.423455] Lustre: Skipped 2 previous similar messages [ 6827.806476] Lustre: server umount lustre-MDT0000 complete [ 6829.169667] LustreError: 254793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859349 with bad export cookie 8563679210986687138 [ 6829.172854] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6829.174112] LustreError: 254793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6829.300437] Lustre: server umount lustre-MDT0001 complete [ 6835.262323] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6838.358364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6838.486596] LustreError: 259304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6838.500621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6838.503042] Lustre: Skipped 2 previous similar messages [ 6839.852490] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6842.785271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6844.289408] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6845.269171] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6847.538558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6849.756774] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6852.690096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6852.756890] Lustre: lustre-OST0001: new disk, initializing [ 6852.758748] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6852.761451] Lustre: 261559:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6854.686931] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6854.720354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6854.723912] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6854.753329] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6859.126236] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6860.370218] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6862.358083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6862.359814] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6864.864906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6864.868047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6864.871839] Lustre: Skipped 2 previous similar messages [ 6864.874523] Lustre: Skipped 2 previous similar messages [ 6870.361591] Lustre: server umount lustre-MDT0000 complete [ 6871.489372] LustreError: 259287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859391 with bad export cookie 8563679210986688027 [ 6871.492184] LustreError: 259287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6871.492898] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6871.597329] Lustre: server umount lustre-MDT0001 complete [ 6882.312295] Lustre: server umount lustre-OST0000 complete [ 6893.797861] Lustre: server umount lustre-OST0001 complete [ 6899.149096] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6902.558074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6902.705769] LustreError: 264591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6902.725748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6902.727709] Lustre: Skipped 3 previous similar messages [ 6904.101937] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6907.030645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6908.521553] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6909.572573] Lustre: 265700:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6911.901340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6913.059038] LustreError: 266053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6913.065794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6913.949419] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6916.880292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6918.001319] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6919.002497] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6922.341986] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6923.632312] Lustre: 267538:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6925.373123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6925.386977] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6928.388069] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:10:48 (1780859448) [ 6943.199918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6943.201917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6943.205168] Lustre: Skipped 1 previous similar message [ 6943.206792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6943.210355] Lustre: Skipped 6 previous similar messages [ 6945.953346] Lustre: server umount lustre-MDT0000 complete [ 6947.139370] LustreError: 264572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859467 with bad export cookie 8563679210986694334 [ 6947.142996] LustreError: 264572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6947.144580] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6948.831954] LustreError: 265316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6948.836400] LustreError: 265316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6953.331061] Lustre: server umount lustre-MDT0001 complete [ 6960.701566] Lustre: server umount lustre-OST0000 complete [ 6968.025792] Lustre: server umount lustre-OST0001 complete [ 6970.052823] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 6973.281437] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6988.590465] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 6992.013544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6992.123899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6992.137267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6992.177829] Lustre: lustre-MDT0000: new disk, initializing [ 6992.209972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6993.557067] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 6997.708415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6997.755862] Lustre: 271717:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6997.864708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6999.181770] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7003.603944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7003.716744] Lustre: 273316:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7004.830743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7005.655331] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7009.814449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7011.631968] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7011.803122] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7016.193298] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7021.429311] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 15:12:21 (1780859541) [ 7023.584230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7023.587982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7023.594434] Lustre: Skipped 5 previous similar messages [ 7023.597662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7023.600198] Lustre: Skipped 7 previous similar messages [ 7028.259196] Lustre: server umount lustre-MDT0000 complete [ 7029.543590] LustreError: 272495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859549 with bad export cookie 8563679210986702209 [ 7029.546947] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7029.547799] LustreError: 272495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7029.670483] Lustre: server umount lustre-MDT0001 complete [ 7041.548475] Lustre: server umount lustre-OST0000 complete [ 7052.067857] Lustre: server umount lustre-OST0001 complete [ 7054.895829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7055.064192] LustreError: 276395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7055.073311] LustreError: 276395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7055.097936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7055.100042] Lustre: Skipped 7 previous similar messages [ 7056.480418] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7059.222525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.664124] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7062.410958] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7063.641514] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7065.938810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7068.035179] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7069.833921] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7074.275645] LustreError: 277996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7074.280146] LustreError: 277996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7091.351519] loop1: detected capacity change from 0 to 204800000 [ 7091.455968] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7091.462073] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7091.468181] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7093.163350] loop1: detected capacity change from 0 to 204800000 [ 7095.068561] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7095.107768] LustreError: 276384:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs15810: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7095.112831] LustreError: 276384:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs15810 [ 7095.114737] LustreError: 276384:0:(mgs_handler.c:517:mgs_target_reg()) fs15810-MDT0001: Error checking target index: rc = -13 [ 7095.118084] LustreError: fs15810-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7095.121717] LustreError: 279797:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 7095.124287] LustreError: 279797:0:(tgt_mount.c:2084:server_put_super()) no obd fs15810-MDT0001 [ 7095.127032] LustreError: 279797:0:(tgt_mount.c:116:server_deregister_mount()) fs15810-MDT0001 not registered [ 7095.139954] Lustre: server umount fs15810-MDT0001 complete [ 7095.141911] LustreError: 279797:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7097.950705] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7097.982533] Lustre: Setting parameter fs15810-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15810-MDT0001 [ 7097.988436] Lustre: Skipped 2 previous similar messages [ 7098.008714] Lustre: srv-fs15810-MDT0001: No data found on store. Initialize space. [ 7098.012021] Lustre: Skipped 4 previous similar messages [ 7098.059217] Lustre: fs15810-MDT0001: new disk, initializing [ 7098.060765] Lustre: Skipped 3 previous similar messages [ 7098.099967] Lustre: srv-fs15810-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7098.102726] Lustre: 280262:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15810-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7098.105763] LustreError: 280262:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15810-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7099.424471] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7102.336589] Lustre: server umount fs15810-MDT0001 complete [ 7106.624231] loop1: detected capacity change from 0 to 204800000 [ 7108.527019] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7108.651211] Lustre: 281809:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15810-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7108.654048] LustreError: 281809:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15810-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7109.955306] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7110.931702] LustreError: 278562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7110.939111] LustreError: 278562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 7111.502156] Lustre: server umount fs15810-MDT0001 complete [ 7115.743951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7115.746183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7115.749757] Lustre: Skipped 3 previous similar messages [ 7120.807580] Lustre: server umount lustre-MDT0000 complete [ 7123.654349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7123.698836] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7125.057299] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7127.521390] loop1: detected capacity change from 0 to 204800000 [ 7129.062346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7129.334363] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7129.449920] Lustre: 284219:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15810-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7129.452973] LustreError: 284219:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15810-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7130.704716] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7132.367304] Lustre: server umount fs15810-MDT0001 complete [ 7139.300592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7145.102477] Lustre: server umount lustre-OST0000 complete [ 7152.476543] Lustre: server umount lustre-MDT0000 complete [ 7153.737471] LustreError: 282784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859673 with bad export cookie 8563679210986704890 [ 7153.741398] LustreError: 282784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7153.852358] Lustre: server umount lustre-MDT0001 complete [ 7157.536385] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7158.618459] Key type lgssc unregistered [ 7158.759372] LNet: 286523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7158.762713] LNetError: 286523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7158.770479] LNet: Removed LNI 192.168.204.151@tcp [ 7159.096125] Key type .llcrypt unregistered [ 7159.097231] Key type ._llcrypt unregistered [ 7166.845970] Key type ._llcrypt registered [ 7166.847608] Key type .llcrypt registered [ 7166.892719] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 7167.293994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7167.347131] alg: No test for adler32 (adler32-zlib) [ 7168.217374] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 7168.318509] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 7169.903150] Key type lgssc registered [ 7170.288073] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7172.976924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7172.984911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7174.149662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7175.441878] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7178.259541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7178.406233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7179.668181] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7181.416710] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7182.595684] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7184.814959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7184.867081] LustreError: 289356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7184.871863] LustreError: 289356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7184.965757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7186.798225] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7188.069682] LustreError: 289357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7188.077271] LustreError: 289357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7188.590452] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7189.744574] LustreError: 289357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7193.573125] LustreError: 289356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7193.580823] LustreError: 289356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7195.490899] loop1: detected capacity change from 0 to 204800000 [ 7195.606886] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7195.610578] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7195.614537] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7197.341039] loop1: detected capacity change from 0 to 204800000 [ 7198.691243] LustreError: 289865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7198.698745] LustreError: 289865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7199.342505] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7199.402432] Lustre: fs14583-OST0000: new disk, initializing [ 7199.404799] Lustre: srv-fs14583-OST0000: No data found on store. Initialize space. [ 7199.407454] Lustre: 291146:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7199.432453] Lustre: fs14583-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7201.379405] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7204.821389] Lustre: server umount fs14583-OST0000 complete [ 7208.919297] loop1: detected capacity change from 0 to 204800000 [ 7208.934513] LustreError: 289356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7208.939775] LustreError: 289356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7210.895732] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7210.969252] Lustre: fs14583-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7212.904412] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7215.785534] loop2: detected capacity change from 0 to 204800000 [ 7215.862194] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7215.865299] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7215.869453] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7217.482502] loop2: detected capacity change from 0 to 204800000 [ 7219.326497] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7219.358573] LustreError: fs14583-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 7219.364876] LustreError: fs14583-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7219.369470] LustreError: 294150:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 7219.372178] LustreError: 294150:0:(tgt_mount.c:2084:server_put_super()) no obd fs14583-OST0001 [ 7219.374580] LustreError: 294150:0:(tgt_mount.c:116:server_deregister_mount()) fs14583-OST0001 not registered [ 7219.386479] Lustre: server umount fs14583-OST0001 complete [ 7219.387918] LustreError: 294150:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7222.825020] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7222.871409] Lustre: fs14583-OST0001: new disk, initializing [ 7222.872914] Lustre: srv-fs14583-OST0001: No data found on store. Initialize space. [ 7222.875178] Lustre: 294700:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7222.896221] Lustre: fs14583-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7224.851436] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7225.619734] LustreError: 289865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7225.624584] LustreError: 289865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 7228.371754] Lustre: server umount fs14583-OST0000 complete [ 7231.476690] Lustre: server umount fs14583-OST0001 complete [ 7235.620486] loop1: detected capacity change from 0 to 204800000 [ 7237.580653] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7239.635624] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7241.863100] loop2: detected capacity change from 0 to 204800000 [ 7243.772881] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7243.838161] Lustre: fs14583-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7243.839963] Lustre: Skipped 1 previous similar message [ 7245.727654] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7248.656293] Lustre: server umount fs14583-OST0000 complete [ 7251.700682] Lustre: server umount fs14583-OST0001 complete [ 7260.130853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7260.131245] LustreError: 289358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7260.131613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7260.132867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7260.135822] Lustre: Skipped 1 previous similar message [ 7260.145097] LustreError: 289358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 7261.522188] Lustre: server umount lustre-OST0000 complete [ 7265.248926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7265.249759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7265.255192] Lustre: Skipped 1 previous similar message [ 7265.257662] Lustre: Skipped 1 previous similar message [ 7268.952950] Lustre: server umount lustre-MDT0000 complete [ 7270.217510] LustreError: 294699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859790 with bad export cookie 11490503011917022008 [ 7270.221339] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7270.222867] LustreError: 294699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7270.357368] Lustre: server umount lustre-MDT0001 complete [ 7273.799747] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7274.790282] Key type lgssc unregistered [ 7274.914455] LNet: 300540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7274.917210] LNetError: 300540:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7274.923400] LNet: Removed LNI 192.168.204.151@tcp [ 7275.244137] Key type .llcrypt unregistered [ 7275.246082] Key type ._llcrypt unregistered [ 7284.942088] Key type ._llcrypt registered [ 7284.943050] Key type .llcrypt registered [ 7284.980365] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 7291.216403] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 7291.633364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7291.696596] alg: No test for adler32 (adler32-zlib) [ 7292.566643] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 7292.664642] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 7294.255174] Key type lgssc registered [ 7294.629886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7305.874446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7305.883067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7306.984077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7306.996085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7307.031724] Lustre: lustre-MDT0000: new disk, initializing [ 7307.059875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7307.069750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7308.479813] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7312.744218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.784703] Lustre: 304336:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7312.808833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7312.810580] Lustre: Skipped 1 previous similar message [ 7312.853935] Lustre: lustre-MDT0001: new disk, initializing [ 7312.891051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7312.908827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7312.912503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7314.265573] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7317.356761] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7318.569876] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7320.805382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7320.923783] Lustre: lustre-OST0000: new disk, initializing [ 7320.925499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7320.927915] Lustre: 306077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7320.957800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7322.301587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7322.305946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7322.341159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7322.756376] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7325.851408] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7326.498368] Lustre: Mounted lustre-client [ 7326.830207] Lustre: Unmounted lustre-client [ 7331.810064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7331.811165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7331.813919] Lustre: Skipped 1 previous similar message [ 7335.696251] Lustre: server umount lustre-OST0000 complete [ 7338.463881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7338.466140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7338.469931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7338.472804] Lustre: Skipped 1 previous similar message [ 7342.048186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7342.048924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7347.168019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7347.170119] Lustre: Skipped 2 previous similar messages [ 7351.263122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7351.264465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7351.267680] Lustre: Skipped 1 previous similar message [ 7351.325177] Lustre: server umount lustre-MDT0000 complete [ 7352.513396] LustreError: 304327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859872 with bad export cookie 4475431145796585510 [ 7352.516661] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7352.518950] LustreError: 304327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7352.639825] Lustre: server umount lustre-MDT0001 complete [ 7355.958198] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7356.986258] Key type lgssc unregistered [ 7385.823419] LNet: 308104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7385.827402] LNetError: 308104:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7456.485306] LNet: Removed LNI 192.168.204.151@tcp [ 7456.756819] Key type .llcrypt unregistered [ 7456.757875] Key type ._llcrypt unregistered [ 7462.536196] Key type ._llcrypt registered [ 7462.537477] Key type .llcrypt registered [ 7462.572379] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 7468.669686] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 7468.992075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7468.998804] alg: No test for adler32 (adler32-zlib) [ 7469.862234] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 7469.950438] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 7471.535156] Key type lgssc registered [ 7471.898945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7482.014665] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:20:01 (1780860001) [ 7484.146754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7484.151702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7485.250539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7485.261454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7485.299483] Lustre: lustre-MDT0000: new disk, initializing [ 7485.323866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7485.329882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7486.641802] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7490.697394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7490.747860] Lustre: 311266:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7490.773736] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7490.775307] Lustre: Skipped 1 previous similar message [ 7490.809346] Lustre: lustre-MDT0001: new disk, initializing [ 7490.841684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7490.857091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7490.862444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7492.148217] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7495.155119] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7496.414822] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7498.681223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7498.797870] Lustre: lustre-OST0000: new disk, initializing [ 7498.799679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7498.802599] Lustre: 313007:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7498.835660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7500.641448] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7503.551777] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7504.377527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7504.382529] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7504.410995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7514.594673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7514.595180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7514.596724] LustreError: Skipped 1 previous similar message [ 7514.603522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7514.605763] Lustre: Skipped 1 previous similar message [ 7518.548173] Lustre: server umount lustre-OST0000 complete [ 7521.760297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7521.762425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7521.765533] Lustre: Skipped 1 previous similar message [ 7521.767325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7524.832304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7524.832836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7525.979252] Lustre: server umount lustre-MDT0000 complete [ 7527.189114] LustreError: 312044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780860047 with bad export cookie 16276941122532125404 [ 7527.192021] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7527.195834] LustreError: 312044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7527.321509] Lustre: server umount lustre-MDT0001 complete [ 7530.631473] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7531.705669] Key type lgssc unregistered [ 7531.829523] LNet: 314635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7531.831817] LNetError: 314635:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7531.840409] LNet: Removed LNI 192.168.204.151@tcp [ 7532.169524] Key type .llcrypt unregistered [ 7532.170722] Key type ._llcrypt unregistered [ 7541.355792] Key type ._llcrypt registered [ 7541.357552] Key type .llcrypt registered [ 7541.395608] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 7541.752831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7541.774071] alg: No test for adler32 (adler32-zlib) [ 7542.624035] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 7542.710220] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 7544.287187] Key type lgssc registered [ 7544.653154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7547.228689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7547.236390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7548.406134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7549.729096] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7552.424045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7552.542152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7553.763308] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7555.520701] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7556.603669] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7558.703639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7558.806886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7560.539804] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7562.210394] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7580.626907] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7581.154283] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7583.002113] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.51@tcp (stopping) [ 7583.199875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7583.201859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7583.714387] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7583.715297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7583.720935] Lustre: Skipped 1 previous similar message [ 7583.723865] Lustre: Skipped 1 previous similar message [ 7584.606691] Lustre: server umount lustre-MDT0000 complete [ 7585.890454] LustreError: 315994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780860105 with bad export cookie 5518284776569943079 [ 7585.895939] LustreError: 315994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7585.896247] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7588.011526] Lustre: server umount lustre-MDT0001 complete [ 7605.590492] Lustre: server umount lustre-OST0000 complete [ 7608.254842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7609.476827] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7610.342091] Lustre: 319760:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.151@tcp [ 7611.015550] Lustre: server umount MGS complete [ 7613.775412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7613.921233] LustreError: 320331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7613.926294] LustreError: 320331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7613.942773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7614.675212] LustreError: 320328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7615.195404] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7617.875931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7618.000161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7619.338547] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7622.189137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7624.160393] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7625.920472] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7626.489140] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7627.707878] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7628.266133] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7632.865020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7632.866322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7632.868595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7632.871503] Lustre: Skipped 1 previous similar message [ 7632.875843] Lustre: Skipped 2 previous similar messages [ 7635.465359] Lustre: server umount lustre-OST0000 complete [ 7637.984980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7637.985459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7637.989746] Lustre: Skipped 1 previous similar message [ 7637.992647] Lustre: Skipped 1 previous similar message [ 7642.844844] Lustre: server umount lustre-MDT0000 complete [ 7643.104507] LustreError: 320350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7643.110385] LustreError: 320350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7644.109206] LustreError: 320314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780860164 with bad export cookie 5518284776569943905 [ 7644.111940] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7644.114749] LustreError: 320314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7644.239175] Lustre: server umount lustre-MDT0001 complete [ 7647.642405] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7648.673853] Key type lgssc unregistered [ 7648.818373] LNet: 323502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7648.822134] LNetError: 323502:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7648.835292] LNet: Removed LNI 192.168.204.151@tcp [ 7649.143501] Key type .llcrypt unregistered [ 7649.144508] Key type ._llcrypt unregistered [ 7655.087370] Key type ._llcrypt registered [ 7655.088437] Key type .llcrypt registered [ 7655.127817] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 7661.226525] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 7661.532644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7661.562327] alg: No test for adler32 (adler32-zlib) [ 7662.427773] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 7662.503996] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 7664.079102] Key type lgssc registered [ 7664.402388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7675.445216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7675.450143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7676.534718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7676.545314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7676.579798] Lustre: lustre-MDT0000: new disk, initializing [ 7676.602046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.608485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7677.869454] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7682.421185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.463614] Lustre: 326598:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7682.490156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7682.493311] Lustre: Skipped 1 previous similar message [ 7682.533155] Lustre: lustre-MDT0001: new disk, initializing [ 7682.568338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7682.581554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7682.584707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7683.897990] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7687.018657] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7688.290223] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7690.512744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7690.612591] Lustre: lustre-OST0000: new disk, initializing [ 7690.614441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7690.617050] Lustre: 328340:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7690.645192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7692.408441] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7692.860440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7692.865233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7692.903815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7695.864146] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7697.459805] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7697.535790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7698.974324] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7699.076466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7703.010953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7703.011744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7703.011759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7703.014416] Lustre: Skipped 1 previous similar message [ 7708.130389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7708.132940] Lustre: Skipped 2 previous similar messages [ 7713.250547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7713.253418] Lustre: Skipped 1 previous similar message [ 7714.271183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7714.350580] Lustre: server umount lustre-OST0000 complete [ 7718.369085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7718.369317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7718.369867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7718.372436] Lustre: Skipped 1 previous similar message [ 7721.750722] Lustre: server umount lustre-MDT0000 complete [ 7723.038384] LustreError: 326593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780860243 with bad export cookie 14110901905303747218 [ 7723.042931] LustreError: 326593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7723.045158] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7723.165505] Lustre: server umount lustre-MDT0001 complete [ 7725.950512] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 15:24:05 (1780860245) [ 7731.117907] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 7734.323345] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 7745.294824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7745.381467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7745.392479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7745.428713] Lustre: lustre-MDT0000: new disk, initializing [ 7745.450205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7745.456538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7746.666157] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7750.647398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7750.683310] Lustre: 332911:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7750.707070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7750.710526] Lustre: Skipped 1 previous similar message [ 7750.745419] Lustre: lustre-MDT0001: new disk, initializing [ 7750.773672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7750.783528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7750.786647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7751.984231] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7754.947236] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7756.107232] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7758.326232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7758.425139] Lustre: 334652:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7760.247759] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 7763.289632] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7763.450707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7763.455501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7763.482463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7778.786571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7778.786962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7778.791139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7778.792219] LustreError: Skipped 1 previous similar message [ 7778.793906] Lustre: Skipped 1 previous similar message [ 7783.631461] Lustre: server umount lustre-OST0000 complete [ 7784.807821] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7784.810265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7784.814131] Lustre: Skipped 1 previous similar message [ 7789.024826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7789.025645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7789.028455] Lustre: Skipped 1 previous similar message [ 7799.263097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7799.322027] Lustre: server umount lustre-MDT0000 complete [ 7800.484130] LustreError: 332904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780860320 with bad export cookie 14110901905303748086 [ 7800.488187] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7800.489875] LustreError: 332904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7800.604506] Lustre: server umount lustre-MDT0001 complete [ 7803.831398] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7804.799882] Key type lgssc unregistered [ 7804.916402] LNet: 336677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7804.919214] LNetError: 336677:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7804.928305] LNet: Removed LNI 192.168.204.151@tcp [ 7805.202809] Key type .llcrypt unregistered [ 7805.203890] Key type ._llcrypt unregistered [ 7835.780531] Key type ._llcrypt registered [ 7835.782121] Key type .llcrypt registered [ 7835.818249] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7835.909432] Key type .llcrypt unregistered [ 7835.910579] Key type ._llcrypt unregistered [ 7861.963713] Key type ._llcrypt registered [ 7861.965480] Key type .llcrypt registered [ 7862.001616] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7862.096218] Key type .llcrypt unregistered [ 7862.097439] Key type ._llcrypt unregistered [ 7873.892650] Key type ._llcrypt registered [ 7873.893755] Key type .llcrypt registered [ 7873.932803] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7874.034272] Key type .llcrypt unregistered [ 7874.035303] Key type ._llcrypt unregistered [ 7885.292195] Key type ._llcrypt registered [ 7885.293476] Key type .llcrypt registered [ 7885.334463] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7885.436236] Key type .llcrypt unregistered [ 7885.437675] Key type ._llcrypt unregistered [ 7913.442401] Key type ._llcrypt registered [ 7913.443483] Key type .llcrypt registered [ 7913.477710] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7913.568480] Key type .llcrypt unregistered [ 7913.569617] Key type ._llcrypt unregistered [ 7924.048097] Key type ._llcrypt registered [ 7924.049248] Key type .llcrypt registered [ 7924.084892] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7924.183563] Key type .llcrypt unregistered [ 7924.184725] Key type ._llcrypt unregistered [ 7936.678689] Key type ._llcrypt registered [ 7936.679781] Key type .llcrypt registered [ 7936.715581] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7936.812285] Key type .llcrypt unregistered [ 7936.813384] Key type ._llcrypt unregistered [ 7947.627513] Key type ._llcrypt registered [ 7947.628536] Key type .llcrypt registered [ 7947.668917] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7947.769379] Key type .llcrypt unregistered [ 7947.770586] Key type ._llcrypt unregistered [ 7957.978119] Key type ._llcrypt registered [ 7957.979194] Key type .llcrypt registered [ 7958.016291] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7958.115488] Key type .llcrypt unregistered [ 7958.116568] Key type ._llcrypt unregistered [ 7968.656389] Key type ._llcrypt registered [ 7968.657577] Key type .llcrypt registered [ 7968.701076] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7968.808207] Key type .llcrypt unregistered [ 7968.809234] Key type ._llcrypt unregistered [ 7990.783077] Key type ._llcrypt registered [ 7990.784140] Key type .llcrypt registered [ 7990.818639] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 7990.923221] Key type .llcrypt unregistered [ 7990.924493] Key type ._llcrypt unregistered [ 8002.649096] Key type ._llcrypt registered [ 8002.650156] Key type .llcrypt registered [ 8002.690418] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 8002.830253] Key type .llcrypt unregistered [ 8002.832430] Key type ._llcrypt unregistered [ 8014.593393] Key type ._llcrypt registered [ 8014.594612] Key type .llcrypt registered [ 8014.629735] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 8014.717529] Key type .llcrypt unregistered [ 8014.718593] Key type ._llcrypt unregistered [ 8035.666167] Key type ._llcrypt registered [ 8035.667625] Key type .llcrypt registered [ 8035.711259] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 8042.211207] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 8042.637855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8042.649519] alg: No test for adler32 (adler32-zlib) [ 8043.501389] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 8043.580042] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 8045.151153] Key type lgssc registered [ 8045.606579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8056.104967] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8056.109700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8057.199257] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8057.209464] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8057.246773] Lustre: tst250fs-MDT0000: new disk, initializing [ 8057.269485] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8057.275845] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8058.498533] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8062.482331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8062.521119] Lustre: 356327:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8062.544990] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8062.546776] Lustre: Skipped 1 previous similar message [ 8062.582191] Lustre: tst250fs-MDT0001: new disk, initializing [ 8062.611955] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8062.625152] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8062.628487] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8063.850914] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8066.018114] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8068.065675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8068.172874] Lustre: tst250fs-OST0000: new disk, initializing [ 8068.175567] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8068.178538] Lustre: 357925:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8068.211278] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8070.003302] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8074.004417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8074.060080] Lustre: tst250fs-OST0001: new disk, initializing [ 8074.061665] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8074.064088] Lustre: 358768:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8074.093725] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8075.208100] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8075.212612] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8075.272492] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8075.278776] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8075.846106] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug -1 all [ 8078.279623] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8088.543977] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8088.546216] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8088.550032] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8090.595701] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8090.599793] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8090.606733] Lustre: Skipped 2 previous similar messages [ 8090.614337] Lustre: Skipped 3 previous similar messages [ 8094.298123] Lustre: server umount tst250fs-MDT0000 complete [ 8095.479604] LustreError: 359377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780860615 with bad export cookie 2588462096593789802 [ 8095.485530] LustreError: 359377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8095.487861] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8095.598823] Lustre: server umount tst250fs-MDT0001 complete [ 8106.113397] Lustre: server umount tst250fs-OST0000 complete [ 8117.529949] Lustre: server umount tst250fs-OST0001 complete [ 8123.046668] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 8125.411884] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 8134.566699] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 15:30:54 (1780860654) [ 8135.064577] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8140.058182] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 8142.950377] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 8152.678305] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7834 sec ========== 15:31:12 (1780860672) [ 8153.169470] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:31:12 (1780860672) === [ 8154.160863] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:31:13 (1780860673) === [ 8166.687849] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 8167.571628] Key type lgssc unregistered [ 8167.698319] LNet: 367783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8167.700570] LNetError: 367783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8167.713313] LNet: Removed LNI 192.168.204.151@tcp [ 8167.988082] Key type .llcrypt unregistered [ 8167.989187] Key type ._llcrypt unregistered