[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 454205252 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003352] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008794] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010020] pid_max: default: 32768 minimum: 301 [ 0.011175] LSM: Security Framework initializing [ 0.012112] Yama: becoming mindful. [ 0.013067] SELinux: Initializing. [ 0.014109] *** VALIDATE selinux *** [ 0.024938] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029166] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031070] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033116] *** VALIDATE tmpfs *** [ 0.035359] *** VALIDATE proc *** [ 0.036253] *** VALIDATE cgroup *** [ 0.037009] *** VALIDATE cgroup2 *** [ 0.038267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040122] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042032] Spectre V2 : User space: Vulnerable [ 0.043010] Speculative Store Bypass: Vulnerable [ 0.045974] debug: unmapping init [mem 0xffffffffad459000-0xffffffffad460fff] [ 0.047200] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048712] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049028] ... version: 2 [ 0.050015] ... bit width: 48 [ 0.051012] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053017] ... max period: 00007fffffffffff [ 0.054015] ... fixed-purpose events: 3 [ 0.055017] ... event mask: 000000070000000f [ 0.056401] rcu: Hierarchical SRCU implementation. [ 0.058590] smp: Bringing up secondary CPUs ... [ 0.059594] x86: Booting SMP configuration: [ 0.060036] .... node #0, CPUs: #1 #2 #3 [ 0.064061] smp: Brought up 1 node, 4 CPUs [ 0.066017] smpboot: Max logical packages: 1 [ 0.067013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.151568] node 0 deferred pages initialised in 81ms [ 0.155012] devtmpfs: initialized [ 0.156238] x86/mm: Memory block size: 128MB [ 0.158776] gcov: version magic: 0x41383552 [ 0.160298] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162276] pinctrl core: initialized pinctrl subsystem [ 0.163211] [ 0.163685] ************************************************************* [ 0.164013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165016] ** ** [ 0.166014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167011] ** ** [ 0.168013] ** This means that this kernel is built to expose internal ** [ 0.169017] ** IOMMU data structures, which may compromise security on ** [ 0.170012] ** your system. ** [ 0.171013] ** ** [ 0.172013] ** If you see this message and you are not debugging the ** [ 0.173015] ** kernel, report this immediately to your vendor! ** [ 0.174012] ** ** [ 0.175017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176017] ************************************************************* [ 0.177799] NET: Registered protocol family 16 [ 0.178517] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.180066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.181444] cpuidle: using governor menu [ 0.182000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184630] PCI: Using configuration type 1 for base access [ 0.187150] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.197056] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.198042] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.200032] cryptd: max_cpu_qlen set to 1000 [ 0.202290] ACPI: Added _OSI(Module Device) [ 0.203012] ACPI: Added _OSI(Processor Device) [ 0.204012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.209118] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211589] ACPI: Interpreter enabled [ 0.212058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.213009] ACPI: Using IOAPIC for interrupt routing [ 0.214109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.215380] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.225238] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.227021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.230388] acpiphp: Slot [2] registered [ 0.231129] acpiphp: Slot [5] registered [ 0.232117] acpiphp: Slot [6] registered [ 0.233093] acpiphp: Slot [7] registered [ 0.234138] acpiphp: Slot [8] registered [ 0.235123] acpiphp: Slot [9] registered [ 0.236187] acpiphp: Slot [10] registered [ 0.237155] acpiphp: Slot [3] registered [ 0.238095] acpiphp: Slot [4] registered [ 0.239084] acpiphp: Slot [11] registered [ 0.240114] acpiphp: Slot [12] registered [ 0.241085] acpiphp: Slot [13] registered [ 0.242116] acpiphp: Slot [14] registered [ 0.243103] acpiphp: Slot [15] registered [ 0.244138] acpiphp: Slot [16] registered [ 0.245136] acpiphp: Slot [17] registered [ 0.246107] acpiphp: Slot [18] registered [ 0.247086] acpiphp: Slot [19] registered [ 0.248095] acpiphp: Slot [20] registered [ 0.249106] acpiphp: Slot [21] registered [ 0.250130] acpiphp: Slot [22] registered [ 0.251087] acpiphp: Slot [23] registered [ 0.252109] acpiphp: Slot [24] registered [ 0.253099] acpiphp: Slot [25] registered [ 0.254129] acpiphp: Slot [26] registered [ 0.255107] acpiphp: Slot [27] registered [ 0.256097] acpiphp: Slot [28] registered [ 0.257162] acpiphp: Slot [29] registered [ 0.258130] acpiphp: Slot [30] registered [ 0.259103] acpiphp: Slot [31] registered [ 0.260055] PCI host bridge to bus 0000:00 [ 0.260949] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.261032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.262028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.263032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.264030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.265048] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.266233] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.268057] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.270370] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.276022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.280058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.281023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.282027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.283021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.284649] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.285817] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.286044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.287864] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.290018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.300000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.301809] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.305421] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.308019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.311020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.318022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.324030] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.327016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.330017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.337018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.343325] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.346018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.349019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.356024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.362666] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.365016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.368017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.375016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.381059] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.384018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.387019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.394021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.398000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.416017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.423016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.440014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.451620] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.455507] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.458454] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.461482] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.464415] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.469163] iommu: Default domain type: Passthrough [ 0.471475] SCSI subsystem initialized [ 0.474163] ACPI: bus type USB registered [ 0.476135] usbcore: registered new interface driver usbfs [ 0.478092] usbcore: registered new interface driver hub [ 0.481128] usbcore: registered new device driver usb [ 0.483294] pps_core: LinuxPPS API ver. 1 registered [ 0.485016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.489078] PTP clock support registered [ 0.492046] EDAC MC: Ver: 3.0.0 [ 0.494124] PCI: Using ACPI for IRQ routing [ 0.496972] NetLabel: Initializing [ 0.498013] NetLabel: domain hash size = 128 [ 0.500013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.502096] NetLabel: unlabeled traffic allowed by default [ 0.505202] vgaarb: loaded [ 0.506355] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.509016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.520060] clocksource: Switched to clocksource kvm-clock [ 0.621456] VFS: Disk quotas dquot_6.6.0 [ 0.623326] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.627677] *** VALIDATE ramfs *** [ 0.629135] *** VALIDATE hugetlbfs *** [ 0.631055] pnp: PnP ACPI init [ 0.633867] pnp: PnP ACPI: found 6 devices [ 0.661812] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.665539] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.668453] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.671293] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.674256] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.676641] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.678907] NET: Registered protocol family 2 [ 0.681725] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.686098] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.688496] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.692368] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.697427] TCP: Hash tables configured (established 65536 bind 65536) [ 0.701045] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.704517] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.708032] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.711511] NET: Registered protocol family 1 [ 0.714553] RPC: Registered named UNIX socket transport module. [ 0.717216] RPC: Registered udp transport module. [ 0.719311] RPC: Registered tcp transport module. [ 0.721565] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.724449] NET: Registered protocol family 44 [ 0.726070] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.728501] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.730856] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.733437] PCI: CLS 0 bytes, default 64 [ 0.735329] Unpacking initramfs... [ 2.088970] debug: unmapping init [mem 0xffff9a217cc54000-0xffff9a217ffbffff] [ 2.092484] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.093985] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.096847] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.616783] Initialise system trusted keyrings [ 2.618517] Key type blacklist registered [ 2.620412] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.631524] zbud: loaded [ 2.634457] *** VALIDATE nfs *** [ 2.635922] *** VALIDATE nfs4 *** [ 2.637434] pstore: using deflate compression [ 2.640973] Platform Keyring initialized [ 2.742767] NET: Registered protocol family 38 [ 2.744276] Key type asymmetric registered [ 2.745914] Asymmetric key parser 'x509' registered [ 2.747549] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.750207] io scheduler mq-deadline registered [ 2.751766] io scheduler kyber registered [ 2.752995] io scheduler bfq registered [ 2.754168] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.756902] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.759258] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.761739] ACPI: Power Button [PWRF] [ 2.766945] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.772464] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.783212] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.789897] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.803476] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.829640] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.856869] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.862373] Non-volatile memory driver v1.3 [ 2.864128] Linux agpgart interface v0.103 [ 2.896792] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 2.900220] vda: detected capacity change from 0 to 74825728 [ 2.916806] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.919986] vdb: detected capacity change from 0 to 1073741824 [ 2.944541] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.947686] vdc: detected capacity change from 0 to 2621440000 [ 2.966536] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.969816] vdd: detected capacity change from 0 to 2621440000 [ 2.987656] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.991158] vde: detected capacity change from 0 to 4294967296 [ 3.011504] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.015118] vdf: detected capacity change from 0 to 4294967296 [ 3.036392] libphy: Fixed MDIO Bus: probed [ 3.045180] usbcore: registered new interface driver usbserial_generic [ 3.047934] usbserial: USB Serial support registered for generic [ 3.050382] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.055327] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.057201] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.059772] mousedev: PS/2 mouse device common for all mice [ 3.063493] rtc_cmos 00:05: RTC can wake from S4 [ 3.066257] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.071374] rtc_cmos 00:05: registered as rtc0 [ 3.071481] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.073370] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.079123] intel_pstate: CPU model not supported [ 3.081541] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.085970] hid: raw HID events driver (C) Jiri Kosina [ 3.088227] usbcore: registered new interface driver usbhid [ 3.090340] usbhid: USB HID core driver [ 3.092103] drop_monitor: Initializing network drop monitor service [ 3.094698] Initializing XFRM netlink socket [ 3.096905] NET: Registered protocol family 10 [ 3.100187] Segment Routing with IPv6 [ 3.101655] NET: Registered protocol family 17 [ 3.103482] mpls_gso: MPLS GSO support [ 3.109621] RAS: Correctable Errors collector initialized. [ 3.112093] AVX version of gcm_enc/dec engaged. [ 3.113784] AES CTR mode by8 optimization enabled [ 3.193098] sched_clock: Marking stable (3193034717, 0)->(4409736459, -1216701742) [ 3.195538] registered taskstats version 1 [ 3.197170] Loading compiled-in X.509 certificates [ 3.198454] zswap: loaded using pool lzo/zbud [ 3.223852] Key type big_key registered [ 3.235173] Key type encrypted registered [ 3.237033] ima: No TPM chip found, activating TPM-bypass! [ 3.239187] ima: Allocated hash algorithm: sha1 [ 3.240317] ima: No architecture policies found [ 3.241617] evm: Initialising EVM extended attributes: [ 3.242694] evm: security.selinux [ 3.243597] evm: security.ima [ 3.244600] evm: security.capability [ 3.245691] evm: HMAC attrs: 0x1 [ 3.247228] rtc_cmos 00:05: setting system clock to 2026-08-24 10:50:57 UTC (1787568657) [ 3.251598] debug: unmapping init [mem 0xffffffffae403000-0xffffffffae5fffff] [ 3.254241] debug: unmapping init [mem 0xffffffffad182000-0xffffffffad458fff] [ 3.263124] Write protecting the kernel read-only data: 28672k [ 3.266331] debug: unmapping init [mem 0xffffffffab803000-0xffffffffab9fffff] [ 3.268391] debug: unmapping init [mem 0xffffffffac114000-0xffffffffac1fffff] [ 3.308241] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.316930] systemd[1]: Detected virtualization kvm. [ 3.318226] systemd[1]: Detected architecture x86-64. [ 3.319526] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.345413] systemd[1]: No hostname configured. [ 3.346637] systemd[1]: Set hostname to . [ 3.348890] random: systemd: uninitialized urandom read (16 bytes read) [ 3.351701] systemd[1]: Initializing machine ID from random generator. [ 3.397821] random: ln: uninitialized urandom read (6 bytes read) [ 3.474386] random: systemd: uninitialized urandom read (16 bytes read) [ 3.476176] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.479225] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.484639] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.147321] device-mapper: uevent: version 1.0.3 [ 4.149748] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy[ 4.876370] virtio_net virtio0 ens2: renamed from eth0 Gatherer Daemon. [ 4.885942] random: fast init done Starting dracut initqueue hook... [ 4.932552] scsi host0: ata_piix [ 4.953307] scsi host1: ata_piix [ 4.955235] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.958041] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.368225] dracut-initqueue[591]: RTNETLINK answers: File exists [ 9.749215] random: crng init done [ 9.750467] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.132063] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.269893] printk: systemd: 21 output lines suppressed due to ratelimiting [ 11.492090] SELinux: Disabled at runtime. [ 11.567199] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.578292] systemd[1]: Detected virtualization kvm. [ 11.580547] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.058217] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.061542] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.066256] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.069149] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.073230] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.082806] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.090197] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ 12.159428] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.476581] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.742288] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.766335] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.848551] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.869597] EDAC sbridge: Ver: 1.1.2 [ 14.321959] Key type dns_resolver registered [ 14.612238] NFS: Registering the id_resolver key type [ 14.614215] Key type id_resolver registered [ 14.615654] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg244-server login: [ 42.561821] libcfs: loading out-of-tree module taints kernel. [ 42.586777] Key type ._llcrypt registered [ 42.588257] Key type .llcrypt registered [ 42.641756] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 52.326120] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 53.253271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.265071] alg: No test for adler32 (adler32-zlib) [ 54.412820] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 54.835044] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 56.479177] Key type lgssc registered [ 57.286471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.817837] hrtimer: interrupt took 3479038 ns [ 78.623866] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 127.254590] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 141.124241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 141.160724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.551961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 142.626863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 142.772727] Lustre: lustre-MDT0000: new disk, initializing [ 142.943549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.000797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 148.199591] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.517875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.704585] Lustre: 6501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 163.757669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 163.763381] Lustre: Skipped 1 previous similar message [ 163.887957] Lustre: lustre-MDT0001: new disk, initializing [ 164.022410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.049357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.066571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 168.433922] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.767336] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 184.123087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 184.448477] Lustre: lustre-OST0000: new disk, initializing [ 184.464481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 184.471588] Lustre: 8439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 184.708781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.915331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 184.926532] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 185.086090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 191.439858] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.339334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.540965] Lustre: lustre-OST0001: new disk, initializing [ 207.544752] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 207.550089] Lustre: 9515:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.622167] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.625619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 213.651227] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 213.764772] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 215.642287] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.170802] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 238.676635] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 246.949561] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing check_logdir /tmp/testlogs/ [ 252.884877] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing yml_node [ 257.911178] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 261.080235] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 263.836410] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 265.694591] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 06:55:17 EDT 2026 [ 286.522308] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 288.489717] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 307.690856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 307.693808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.700202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.264179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.264625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.279702] Lustre: Skipped 1 previous similar message [ 311.297617] Lustre: Skipped 1 previous similar message [ 313.308668] Lustre: server umount lustre-MDT0000 complete [ 318.549474] LustreError: 10330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787568972 with bad export cookie 2152849808698540257 [ 318.566153] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 318.566939] LustreError: 10330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 321.504823] LustreError: 6508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.506595] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.519781] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.519790] Lustre: Skipped 1 previous similar message [ 321.537977] LustreError: 6508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 321.607385] Lustre: Skipped 2 previous similar messages [ 323.569544] LustreError: 12457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.578791] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.598088] LustreError: 12457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 323.629052] Lustre: Skipped 2 previous similar messages [ 325.289542] Lustre: server umount lustre-MDT0001 complete [ 331.420614] Lustre: server umount lustre-OST0000 complete [ 337.090612] Lustre: server umount lustre-OST0001 complete [ 345.462409] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 360.906704] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 399.759023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.077915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 400.120656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 400.238530] Lustre: lustre-MDT0000: new disk, initializing [ 400.424129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 400.443631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 406.242080] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 418.488789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.616857] Lustre: 15091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 418.674411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 418.677647] Lustre: Skipped 1 previous similar message [ 418.765293] Lustre: lustre-MDT0001: new disk, initializing [ 418.924505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 418.957573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 418.978769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 424.059604] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 435.240929] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 441.666031] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 449.852285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 450.283532] Lustre: lustre-OST0000: new disk, initializing [ 450.286266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 450.293631] Lustre: 17062:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 450.408212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.488165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 451.500922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 451.697246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 457.004125] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 468.037884] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 473.875328] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 474.158402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 479.328252] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 479.544564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 482.287071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 482.300657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.322198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 482.330427] Lustre: Skipped 1 previous similar message [ 487.938061] Lustre: server umount lustre-OST0000 complete [ 492.512940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.524029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 492.524502] Lustre: Skipped 2 previous similar messages [ 492.545700] Lustre: Skipped 3 previous similar messages [ 498.607677] Lustre: server umount lustre-MDT0000 complete [ 502.752604] LustreError: 17587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.779323] LustreError: 17587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 503.103464] LustreError: 17061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569157 with bad export cookie 2152849808698541790 [ 503.112263] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 503.127806] LustreError: 17061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 503.824983] Lustre: server umount lustre-MDT0001 complete [ 509.832443] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 06:59:22 (1787569162) [ 533.391931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.502245] LustreError: 19938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.634518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 539.620224] LustreError: 19939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.135506] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 549.876867] LustreError: 19938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.897363] LustreError: 19938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 550.880990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.536743] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 565.077679] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 571.761967] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 574.346439] Lustre: Failing over lustre-MDT0000 [ 574.782519] Lustre: server umount lustre-MDT0000 complete [ 576.419266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 576.435597] LustreError: 19938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.457945] LustreError: 19938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 592.609974] Lustre: 3633:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787569230/real 1787569230] req@ffff9a21fb32df80 x1874401646777216/t0(0) o400->MGC192.168.202.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787569246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 592.645825] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.616382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.894623] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1de0757ea63e8c64 [ 601.909847] Lustre: MGC192.168.202.144@tcp: Connection restored to 0@lo (at 0@lo) [ 602.199137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 602.203149] Lustre: Skipped 1 previous similar message [ 602.295577] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 607.219542] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 607.240759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 607.300872] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 608.734669] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 621.497416] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 627.692924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.694722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 627.709241] Lustre: Skipped 2 previous similar messages [ 627.710225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 627.710229] Lustre: Skipped 2 previous similar messages [ 627.727835] LustreError: Skipped 1 previous similar message [ 632.938348] Lustre: server umount lustre-MDT0000 complete [ 637.271472] LustreError: 19920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569291 with bad export cookie 2152849808698543204 [ 637.288428] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 637.585036] Lustre: server umount lustre-MDT0001 complete [ 649.643187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.010949] LustreError: 23293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.023847] LustreError: 23293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 656.160946] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 666.994879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.632631] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 681.620456] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 688.156870] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 690.517290] Lustre: Failing over lustre-MDT0000 [ 690.824099] Lustre: server umount lustre-MDT0000 complete [ 692.196987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.575163] Lustre: 3634:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787569346/real 1787569346] req@ffff9a20c2417480 x1874401646808832/t0(0) o400->MGC192.168.202.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787569362 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 708.613750] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 712.394956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 714.721560] LustreError: 23294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 714.740943] LustreError: 23294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 717.801710] LustreError: 3632:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9a20c2417480 x1874401646811904/t0(0) o250->MGC192.168.202.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 718.195357] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 723.274479] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 723.433483] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 723.448412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 723.457798] Lustre: Skipped 1 previous similar message [ 723.485865] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 732.429951] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 738.791245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 738.806415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 738.830077] Lustre: Skipped 4 previous similar messages [ 742.870398] Lustre: server umount lustre-MDT0000 complete [ 747.261627] LustreError: 23274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569401 with bad export cookie 2152849808698544142 [ 747.291263] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 747.293680] LustreError: 23274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 747.819459] Lustre: server umount lustre-MDT0001 complete [ 758.078264] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 07:03:30 (1787569410) [ 778.138618] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 791.107722] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 840.032966] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 854.354399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 854.738973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 854.788231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 854.884895] Lustre: lustre-MDT0000: new disk, initializing [ 855.021708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 855.035197] Lustre: Skipped 3 previous similar messages [ 855.101938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 861.160848] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 874.404614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 874.596131] Lustre: 29671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 875.125040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 881.483915] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 893.670609] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 901.533429] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 910.327235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 910.684250] Lustre: 31647:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 912.500770] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 912.523349] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 912.590629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 912.612102] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 917.109806] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 928.258207] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 936.664076] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 958.954425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.955874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 958.959227] Lustre: Skipped 3 previous similar messages [ 958.960067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 958.965799] LustreError: Skipped 1 previous similar message [ 958.967780] Lustre: Skipped 2 previous similar messages [ 962.515844] Lustre: server umount lustre-OST0000 complete [ 967.136221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 972.371772] Lustre: server umount lustre-MDT0000 complete [ 974.308425] LustreError: 29684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 974.330343] LustreError: 29684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 976.233684] LustreError: 29664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569630 with bad export cookie 2152849808698544548 [ 976.237269] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 976.247396] LustreError: 29664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 976.768952] Lustre: server umount lustre-MDT0001 complete [ 988.588125] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 991.558755] Key type lgssc unregistered [ 991.794675] LNet: 33626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.798560] LNetError: 33626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.827970] LNet: Removed LNI 192.168.202.144@tcp [ 992.420606] Key type .llcrypt unregistered [ 992.424915] Key type ._llcrypt unregistered [ 1018.758526] Key type ._llcrypt registered [ 1018.761207] Key type .llcrypt registered [ 1018.927910] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 1020.197252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1020.236112] alg: No test for adler32 (adler32-zlib) [ 1021.260119] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 1021.602548] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 1023.375172] Key type lgssc registered [ 1024.838953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1032.660906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1032.681473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1034.306184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.115160] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1044.466905] LustreError: 35047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1044.480066] LustreError: 35047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1048.348437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.952209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1055.223846] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1064.111728] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1070.927284] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1077.629715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1078.035589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.517242] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1092.687085] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1094.468469] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1110.469539] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 07:09:22 (1787569762) [ 1113.446921] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1116.368909] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1124.154430] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 07:09:36 (1787569776) [ 1133.224545] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 07:09:45 (1787569785) [ 1134.316818] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1144.265651] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1144.273766] Lustre: Skipped 2 previous similar messages [ 1146.744272] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 07:09:58 (1787569798) [ 1148.331721] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1148.336090] Lustre: Skipped 2 previous similar messages [ 1156.841702] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1156.847924] Lustre: Skipped 4 previous similar messages [ 1172.033865] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 07:10:24 (1787569824) [ 1175.290862] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1179.462709] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1189.155788] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 07:10:41 (1787569841) [ 1191.265772] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1191.268921] Lustre: Skipped 6 previous similar messages [ 1204.245763] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 07:10:56 (1787569856) [ 1222.627413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1222.634277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1222.638042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1222.659582] Lustre: Skipped 1 previous similar message [ 1222.670632] Lustre: Skipped 1 previous similar message [ 1225.777903] Lustre: server umount lustre-OST0000 complete [ 1232.869780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.875374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1232.892539] Lustre: Skipped 1 previous similar message [ 1232.924193] Lustre: Skipped 1 previous similar message [ 1236.726903] Lustre: server umount lustre-MDT0000 complete [ 1237.988897] LustreError: 35041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1238.013612] LustreError: 35041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1240.785728] LustreError: 35027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569895 with bad export cookie 14841997248110935461 [ 1240.796082] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1240.802330] LustreError: 35027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1241.117165] Lustre: server umount lustre-MDT0001 complete [ 1254.162401] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 1257.493115] Key type lgssc unregistered [ 1257.892674] LNet: 42916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1257.898193] LNetError: 42916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.918751] LNet: Removed LNI 192.168.202.144@tcp [ 1260.120196] Key type .llcrypt unregistered [ 1260.128726] Key type ._llcrypt unregistered [ 1280.371900] Key type ._llcrypt registered [ 1280.374348] Key type .llcrypt registered [ 1280.460285] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 1297.250467] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 1298.670785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1298.748617] alg: No test for adler32 (adler32-zlib) [ 1299.762307] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 1300.022363] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 1301.679468] Key type lgssc registered [ 1302.953511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1346.946662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1346.987135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.287429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1348.332031] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1348.445241] Lustre: lustre-MDT0000: new disk, initializing [ 1348.562106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1348.577815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1354.097841] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1367.318777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.452415] Lustre: 46071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1367.506777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1367.512768] Lustre: Skipped 1 previous similar message [ 1367.692470] Lustre: lustre-MDT0001: new disk, initializing [ 1367.805097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1367.870703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1367.891160] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1373.430702] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1385.185261] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1392.072151] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1399.422320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1399.722116] Lustre: lustre-OST0000: new disk, initializing [ 1399.725718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1399.750293] Lustre: 48040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1399.936911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.035330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1406.048646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1406.206220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1406.490794] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1416.789334] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1422.160191] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1422.375942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1427.611653] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1427.929339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1432.036724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.050817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1432.058237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1432.070416] Lustre: Skipped 1 previous similar message [ 1437.155720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1437.176657] Lustre: Skipped 2 previous similar messages [ 1442.278052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1442.284121] Lustre: Skipped 1 previous similar message [ 1444.322647] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1444.474534] Lustre: server umount lustre-OST0000 complete [ 1449.953485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1449.958682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.966568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1452.518244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1457.632269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1457.640765] Lustre: Skipped 2 previous similar messages [ 1462.239712] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1462.460532] Lustre: server umount lustre-MDT0000 complete [ 1466.911118] LustreError: 46063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570121 with bad export cookie 10144010126865253944 [ 1466.923513] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1467.268382] Lustre: server umount lustre-MDT0001 complete [ 1479.436192] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 07:15:31 (1787570131) [ 1487.993145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1488.284045] LustreError: 50503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.313894] LustreError: 50503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1488.417704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1492.895281] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1493.475135] LustreError: 50504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.596664] LustreError: 50503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1503.715803] LustreError: 50504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1503.826603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1504.088835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1509.313204] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1517.397196] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1524.522697] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1532.161041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1532.386627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.605574] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1551.109540] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1556.397762] LustreError: 53033:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1557.612810] Lustre: Modifying parameter general.timeout=1 in log params [ 1558.750922] Lustre: Modifying parameter general.timeout=2 in log params [ 1561.135506] Lustre: Modifying parameter general.timeout=4 in log params [ 1561.152331] Lustre: Skipped 1 previous similar message [ 1565.342449] Lustre: Modifying parameter general.timeout=8 in log params [ 1565.347269] Lustre: Skipped 3 previous similar messages [ 1573.964636] Lustre: Modifying parameter general.timeout=16 in log params [ 1573.968785] Lustre: Skipped 7 previous similar messages [ 1590.596149] Lustre: Modifying parameter general.timeout=31 in log params [ 1590.604737] Lustre: Skipped 14 previous similar messages [ 1622.656349] Lustre: Modifying parameter general.timeout=61 in log params [ 1622.659716] Lustre: Skipped 29 previous similar messages [ 1687.292669] Lustre: Modifying parameter general.timeout=120 in log params [ 1687.300904] Lustre: Skipped 58 previous similar messages [ 1711.795344] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 07:19:23 (1787570363) [ 1715.301453] Lustre: 59987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1717.313835] Lustre: 60086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1719.792810] Lustre: 60185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1724.638348] Lustre: 60398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1724.661888] Lustre: 60398:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1760.797645] Lustre: 61870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1760.818869] Lustre: 61870:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1760.862701] LustreError: 61870:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1766.200777] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 07:20:18 (1787570418) [ 1785.824929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1785.848735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.863542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.869177] Lustre: Skipped 1 previous similar message [ 1788.607689] Lustre: server umount lustre-MDT0000 complete [ 1791.475448] LustreError: 51655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.493285] LustreError: 51655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1793.239192] LustreError: 50484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570447 with bad export cookie 10144010126865254826 [ 1793.255626] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1793.259068] LustreError: 50484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1793.949789] Lustre: server umount lustre-MDT0001 complete [ 1809.709490] Lustre: server umount lustre-OST0000 complete [ 1839.697154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.155772] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1840.219945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1840.232696] Lustre: Skipped 32 previous similar messages [ 1840.557348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.921272] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1855.922057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.995900] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1856.053507] Lustre: 64402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1861.234928] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1870.428589] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1879.146733] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1886.528642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1886.669582] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1887.024133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.030971] Lustre: Skipped 1 previous similar message [ 1894.677070] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1903.680428] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1906.261851] Lustre: 66979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1914.854477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1914.867568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.879356] Lustre: Skipped 2 previous similar messages [ 1914.886798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1914.897676] Lustre: Skipped 3 previous similar messages [ 1920.345838] Lustre: server umount lustre-OST0000 complete [ 1925.604144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.622933] Lustre: Skipped 1 previous similar message [ 1931.793630] Lustre: server umount lustre-MDT0000 complete [ 1935.847074] LustreError: 64413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.873253] LustreError: 64413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1936.884271] LustreError: 64393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570591 with bad export cookie 10144010126865258046 [ 1936.885439] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1936.902753] LustreError: 64393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1937.636784] Lustre: server umount lustre-MDT0001 complete [ 1952.723575] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 1956.223640] Key type lgssc unregistered [ 1956.555814] LNet: 68189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.569541] LNetError: 68189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1957.618342] LNet: Removed LNI 192.168.202.144@tcp [ 1958.869621] Key type .llcrypt unregistered [ 1958.878802] Key type ._llcrypt unregistered [ 1990.137883] Key type ._llcrypt registered [ 1990.141596] Key type .llcrypt registered [ 1990.291188] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 1991.581279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1991.679648] alg: No test for adler32 (adler32-zlib) [ 1992.979377] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 1993.180217] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 1994.839833] Key type lgssc registered [ 1996.587545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2006.045540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2006.071460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.807118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.496238] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2017.249337] LustreError: 69606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.290668] LustreError: 69606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2022.375030] LustreError: 69607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.112624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.950060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2029.422416] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2040.028841] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2047.917411] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2056.078654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2056.803516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.645389] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2073.155952] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2085.835368] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2096.612867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2096.621731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.640528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.160017] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.162332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.187655] Lustre: Skipped 1 previous similar message [ 2098.202297] Lustre: Skipped 2 previous similar messages [ 2101.229455] Lustre: server umount lustre-MDT0000 complete [ 2103.269691] LustreError: 69606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.304451] LustreError: 69606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2105.343679] LustreError: 69586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570759 with bad export cookie 11001687318804341556 [ 2105.345085] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2105.359854] LustreError: 69586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2105.715384] Lustre: server umount lustre-MDT0001 complete [ 2121.092603] Lustre: server umount lustre-OST0000 complete [ 2151.092708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.421507] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2151.471733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2151.733483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2156.915715] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2167.410877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.558877] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2167.603208] Lustre: 74480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2168.082417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2174.538598] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2183.714693] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2192.469920] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2201.414743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2201.650732] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2202.052859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.162702] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2219.066514] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2221.278966] Lustre: 77056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2223.482845] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2231.783459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2231.796082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.817094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2234.930771] Lustre: server umount lustre-OST0000 complete [ 2239.456664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2239.467046] LustreError: Skipped 1 previous similar message [ 2239.479430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.500459] Lustre: Skipped 1 previous similar message [ 2239.506780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2239.515155] Lustre: Skipped 2 previous similar messages [ 2245.607366] Lustre: server umount lustre-MDT0000 complete [ 2247.654067] LustreError: 74772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.673673] LustreError: 74772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2250.096542] LustreError: 76347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570904 with bad export cookie 11001687318804342403 [ 2250.098411] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2250.110426] LustreError: 76347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2250.694691] Lustre: server umount lustre-MDT0001 complete [ 2264.546577] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 2267.085058] Key type lgssc unregistered [ 2267.319561] LNet: 78274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2267.332575] LNetError: 78274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2267.350846] LNet: Removed LNI 192.168.202.144@tcp [ 2268.354953] Key type .llcrypt unregistered [ 2268.358506] Key type ._llcrypt unregistered [ 2296.157846] Key type ._llcrypt registered [ 2296.165279] Key type .llcrypt registered [ 2296.308977] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 2297.791821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2297.828296] alg: No test for adler32 (adler32-zlib) [ 2298.976198] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 2299.320695] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 2301.063147] Key type lgssc registered [ 2302.253519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2310.526642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2310.542470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2312.082983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.408892] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2322.403465] LustreError: 79688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.415459] LustreError: 79688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2325.711332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.438120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2331.945412] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2341.146691] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2347.385044] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2355.087504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2356.000769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.281696] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2373.123895] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2379.525551] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2380.575278] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2381.683658] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2383.691902] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2383.697983] Lustre: Skipped 1 previous similar message [ 2388.697524] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2388.703371] Lustre: Skipped 4 previous similar messages [ 2397.263331] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2397.271122] Lustre: Skipped 7 previous similar messages [ 2413.284626] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2413.289784] Lustre: Skipped 15 previous similar messages [ 2445.430467] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2445.433734] Lustre: Skipped 32 previous similar messages [ 2487.619608] Lustre: 87795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2490.546396] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 07:32:22 (1787571142) [ 2492.706392] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2495.114512] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 07:32:27 (1787571147) [ 2501.601599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2501.616453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2501.646574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2504.680257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.686559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2504.691085] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2504.719612] Lustre: Skipped 1 previous similar message [ 2506.114400] Lustre: server umount lustre-OST0000 complete [ 2510.817187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2510.828918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.844793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2514.912262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2514.912883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.344943] Lustre: server umount lustre-MDT0000 complete [ 2520.033115] LustreError: 79684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.061361] LustreError: 79684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2520.549430] LustreError: 79669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571174 with bad export cookie 16552668565618429094 [ 2520.563315] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2520.566063] LustreError: 79669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2521.127700] Lustre: server umount lustre-MDT0001 complete [ 2533.434972] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 2536.858033] Key type lgssc unregistered [ 2537.240416] LNet: 89049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2537.257039] LNetError: 89049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2537.295820] LNet: Removed LNI 192.168.202.144@tcp [ 2538.539536] Key type .llcrypt unregistered [ 2538.542251] Key type ._llcrypt unregistered [ 2543.569690] Key type ._llcrypt registered [ 2543.571748] Key type .llcrypt registered [ 2543.703697] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2548.178697] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules [ 2549.467460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2549.518315] alg: No test for adler32 (adler32-zlib) [ 2550.603230] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 2550.839544] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 2552.529233] Key type lgssc registered [ 2553.552993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2554.750534] LustreError: 90252:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2573.212452] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 2577.338480] LustreError: 90252:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2583.988430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2584.003290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2585.692416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2591.072826] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2594.815753] LustreError: 91277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.845897] LustreError: 91277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2599.926419] LustreError: 91278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2603.054806] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 07:34:14 (1787571254) [ 2605.031325] LustreError: 91277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.961484] LustreError: 91309:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2608.981173] LustreError: 91309:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2608.992767] LustreError: 91309:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 2609.801467] Lustre: server umount lustre-MDT0000 complete [ 2626.824198] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 2631.161253] Key type lgssc unregistered [ 2631.765809] LNet: 92806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2631.785989] LNetError: 92806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2632.808742] LNet: Removed LNI 192.168.202.144@tcp [ 2634.277483] Key type .llcrypt unregistered [ 2634.280746] Key type ._llcrypt unregistered [ 2654.310129] Key type ._llcrypt registered [ 2654.311971] Key type .llcrypt registered [ 2654.439502] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 2655.687525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2655.948552] alg: No test for adler32 (adler32-zlib) [ 2657.024660] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 2657.403768] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 2659.111144] Key type lgssc registered [ 2660.478800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2668.969188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2668.993147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2670.716334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2676.127400] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2680.299056] LustreError: 94025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.332430] LustreError: 94025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2685.413531] LustreError: 94026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.808652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.715665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2693.635827] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2702.520742] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2709.087575] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2717.515320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2718.298707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2724.577507] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2733.262941] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2799.649768] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 07:37:31 (1787571451) [ 2802.290091] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2804.410133] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 07:37:36 (1787571456) [ 2810.336902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2810.348269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.363952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2811.878917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2811.879738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2811.894663] Lustre: Skipped 1 previous similar message [ 2811.909925] Lustre: Skipped 2 previous similar messages [ 2814.487246] Lustre: server umount lustre-MDT0000 complete [ 2816.995212] LustreError: 94974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.022317] LustreError: 94974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2818.849968] LustreError: 94007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571473 with bad export cookie 1840731558919803304 [ 2818.859118] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2818.862260] LustreError: 94007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2819.277442] Lustre: server umount lustre-MDT0001 complete [ 2834.626649] Lustre: server umount lustre-OST0000 complete [ 2848.681329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2849.220685] LustreError: 98346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2849.442201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2854.880714] LustreError: 98347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.342091] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2864.099022] LustreError: 98346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.114741] LustreError: 98346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2865.571353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.872567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2872.760225] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2882.074075] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2889.553951] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2905.366106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2905.569277] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2905.581153] LustreError: 98335:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2905.610414] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2905.615817] LustreError: 100431:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2905.623743] LustreError: 100431:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2905.628484] LustreError: 100431:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2905.691558] Lustre: server umount lustre-OST0000 complete [ 2905.694414] LustreError: 100431:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2917.357574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2917.733098] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2917.743263] LustreError: 98336:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2917.785431] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2917.804517] LustreError: 100899:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2917.821609] LustreError: 100899:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2917.834192] LustreError: 100899:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2917.929529] Lustre: server umount lustre-OST0000 complete [ 2917.937650] LustreError: 100899:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2930.194952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2930.470123] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2930.497620] Lustre: Found index 0 for lustre-OST0000, updating log [ 2930.515635] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2930.628236] Lustre: lustre-OST0000: new disk, initializing [ 2930.634340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2930.644793] Lustre: 101434:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2930.934073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.424266] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2941.280557] Lustre: Failing over lustre-OST0000 [ 2941.421969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2941.427392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2941.469357] Lustre: server umount lustre-OST0000 complete [ 2946.593624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.599328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2946.628905] Lustre: Skipped 1 previous similar message [ 2946.632686] Lustre: Skipped 1 previous similar message [ 2950.753239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2950.764953] Lustre: Skipped 1 previous similar message [ 2956.259121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2956.278550] Lustre: Skipped 1 previous similar message [ 2957.279166] Lustre: 93337:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787571595/real 1787571595] req@000000000ae1babf x1874404375111552/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787571611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2960.351257] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2960.637352] Lustre: server umount lustre-MDT0000 complete [ 2961.377146] LustreError: 98347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.429973] LustreError: 98347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2964.894111] LustreError: 98327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571619 with bad export cookie 1840731558919806048 [ 2964.895856] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2964.911306] LustreError: 98327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2965.259757] Lustre: server umount lustre-MDT0001 complete [ 2976.263367] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 07:40:28 (1787571628) [ 3000.002786] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 3017.505463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.554554] LustreError: 103797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3018.855749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3024.554390] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3034.810757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.730653] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3044.570734] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3052.609459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3053.611469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3053.614737] Lustre: Skipped 1 previous similar message [ 3053.641646] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3060.791971] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3060.812840] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3060.821219] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3061.753890] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3073.378796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3073.501659] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3073.679518] Lustre: lustre-OST0001: new disk, initializing [ 3073.696648] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3073.706778] Lustre: 106120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3083.694980] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3084.099236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3084.120590] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3084.349595] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3093.134260] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3099.886578] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3107.297976] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3107.314671] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.338111] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3112.439446] Lustre: server umount lustre-MDT0001 complete [ 3112.979507] LustreError: 106112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3113.007392] LustreError: 106112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3127.971825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3128.063797] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3128.071543] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3128.147511] Lustre: 103785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3129.173640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3129.186969] Lustre: Skipped 1 previous similar message [ 3129.521745] LustreError: 107644:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3129.535742] LustreError: 107644:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3129.567349] LustreError: 103789:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3129.802356] LustreError: 107661:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.144@tcp: cfg command failed: rc = -17 [ 3129.810417] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3129.810417] [ 3134.459592] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3135.299589] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3147.203259] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 07:43:19 (1787571799) [ 3166.610707] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 3187.908732] Lustre: 109734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3214.934227] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3219.852465] Lustre: 110871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3313.633376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.638710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3313.654848] Lustre: Skipped 3 previous similar messages [ 3313.669834] Lustre: Skipped 6 previous similar messages [ 3315.892063] Lustre: server umount lustre-MDT0000 complete [ 3318.242907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3318.261439] LustreError: 103798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3318.289838] LustreError: 103798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 3320.161289] LustreError: 103778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571974 with bad export cookie 1840731558919807049 [ 3320.167118] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3320.178169] LustreError: 103778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3320.480680] Lustre: server umount lustre-MDT0001 complete [ 3334.948920] Lustre: server umount lustre-OST0000 complete [ 3340.260800] Lustre: 93339:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787571978/real 1787571978] req@ffff9a20cd017100 x1874404375385216/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787571994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3340.345908] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.380064] Lustre: Skipped 2 previous similar messages [ 3341.660223] Lustre: server umount lustre-OST0001 complete [ 3350.864697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3364.547955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3375.569661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3388.121150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3411.283064] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 3424.097518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3424.138285] Lustre: lustre-MDT0000: reset Object Index mappings [ 3424.885686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.562653] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3441.629626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3441.651985] Lustre: lustre-MDT0001: reset Object Index mappings [ 3448.397203] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3452.215138] Lustre: 115972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3461.460382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3466.940286] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3468.987550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3470.868570] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3479.510921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3485.269144] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3485.289993] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3489.712731] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3499.422814] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3503.631373] Lustre: 117850:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3563.881317] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 07:50:16 (1787572216) [ 3590.624591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3590.641327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3590.661105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3596.735284] Lustre: server umount lustre-MDT0000 complete [ 3597.794665] LustreError: 114824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.816929] LustreError: 114824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 3601.124097] LustreError: 116353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572255 with bad export cookie 1840731558919853935 [ 3601.146868] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3601.150299] LustreError: 116353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3607.856738] Lustre: server umount lustre-MDT0001 complete [ 3618.750571] Lustre: server umount lustre-OST0000 complete [ 3629.835912] Lustre: server umount lustre-OST0001 complete [ 3637.326258] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 3649.229115] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 3694.282868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3694.544044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3694.574497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3694.673277] Lustre: lustre-MDT0000: new disk, initializing [ 3694.777697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3694.787573] Lustre: Skipped 3 previous similar messages [ 3694.799979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3700.485465] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3707.991558] Lustre: server umount lustre-MDT0000 complete [ 3719.591357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3719.977139] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3720.011968] Lustre: 122122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3720.033513] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3725.489982] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3731.406077] Lustre: server umount lustre-MDT0000 complete [ 3740.830810] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 07:53:13 (1787572393) [ 3742.360731] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3744.377224] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 07:53:16 (1787572396) [ 3763.423320] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 3775.018954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.695725] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3792.761119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3793.020722] Lustre: 124176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3793.176493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3793.190128] Lustre: Skipped 1 previous similar message [ 3793.505257] Lustre: lustre-MDT0001: new disk, initializing [ 3793.805267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3793.821409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3800.289788] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3804.681338] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3811.689298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3812.016641] Lustre: lustre-OST0000: new disk, initializing [ 3812.021818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3812.035841] Lustre: 125753:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3813.706992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3813.717105] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3813.890377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3819.869709] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3831.100402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3831.300644] Lustre: lustre-OST0001: new disk, initializing [ 3831.304779] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3831.310759] Lustre: 126624:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3833.099848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3833.117125] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3833.285266] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3838.700740] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3849.226341] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3853.021506] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4284.939239] Lustre: 133060:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4284.954607] Lustre: 133060:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4284.962359] Lustre: 133060:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4284.968424] Lustre: 133060:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4284.976389] Lustre: 133060:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4284.981568] Lustre: 133060:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4584.920804] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:07:16 (1787573236) [ 4601.320391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4601.325428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4601.338245] Lustre: Skipped 8 previous similar messages [ 4601.342282] Lustre: Skipped 10 previous similar messages [ 4606.437489] LustreError: 124182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4606.450665] LustreError: 124182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4606.476070] Lustre: server umount lustre-MDT0000 complete [ 4610.954798] LustreError: 133074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787573265 with bad export cookie 1840731558919924096 [ 4610.962396] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4610.969657] LustreError: 133074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4611.559065] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.589348] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4611.591884] Lustre: Skipped 2 previous similar messages [ 4625.378048] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4625.971906] Lustre: server umount lustre-MDT0001 complete [ 4645.343187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4645.460286] Lustre: server umount lustre-OST0000 complete [ 4656.815645] Lustre: server umount lustre-OST0001 complete [ 4663.571411] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 4672.959524] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 4711.977991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4712.332637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4712.372783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4712.459575] Lustre: lustre-MDT0000: new disk, initializing [ 4712.544984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4712.554866] Lustre: Skipped 5 previous similar messages [ 4712.571958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4717.795735] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4727.818353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.940523] Lustre: 136123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4727.993681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4727.998268] Lustre: Skipped 1 previous similar message [ 4728.065247] Lustre: lustre-MDT0001: new disk, initializing [ 4728.181046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4728.195405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4732.554570] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4743.198323] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4749.449502] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4755.827882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4756.203224] Lustre: lustre-OST0000: new disk, initializing [ 4756.214281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4756.220783] Lustre: 138092:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4758.300872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4758.316280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4758.430854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4762.860819] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4772.645952] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4783.378993] Lustre: lustre-MDD0000: changelog on [ 4786.380770] Lustre: lustre-MDD0001: changelog on [ 6515.653077] Lustre: lustre-MDD0001: changelog off [ 6517.277211] Lustre: 142307:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6518.150777] Lustre: lustre-MDD0000: changelog off [ 6524.902391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6524.906708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6524.926137] Lustre: Skipped 1 previous similar message [ 6524.933689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6524.933698] Lustre: Skipped 6 previous similar messages [ 6524.963431] LustreError: Skipped 1 previous similar message [ 6530.019985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6530.030212] Lustre: Skipped 2 previous similar messages [ 6530.352928] Lustre: server umount lustre-OST0000 complete [ 6535.141923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.160708] Lustre: Skipped 1 previous similar message [ 6535.164805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6540.217562] Lustre: server umount lustre-MDT0000 complete [ 6540.259587] LustreError: 136136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6540.275473] LustreError: 136136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6543.626389] LustreError: 138091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575197 with bad export cookie 1840731558920742592 [ 6543.626883] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6543.637969] LustreError: 138091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6544.054720] Lustre: server umount lustre-MDT0001 complete [ 6555.898248] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 6558.203326] Key type lgssc unregistered [ 6558.434605] LNet: 147201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6558.438351] LNetError: 147201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6558.448560] LNet: Removed LNI 192.168.202.144@tcp [ 6559.119237] Key type .llcrypt unregistered [ 6559.121146] Key type ._llcrypt unregistered [ 6579.269406] Key type ._llcrypt registered [ 6579.272646] Key type .llcrypt registered [ 6579.382957] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 6597.578943] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 6599.035537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6599.089724] alg: No test for adler32 (adler32-zlib) [ 6600.241487] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 6600.594025] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 6602.279592] Key type lgssc registered [ 6603.259420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6645.666057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6645.698924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.070924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6647.147939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6647.249304] Lustre: lustre-MDT0000: new disk, initializing [ 6647.321099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6647.332807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6652.060619] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6663.463894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.595684] Lustre: 150405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6663.648513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6663.655918] Lustre: Skipped 1 previous similar message [ 6663.746347] Lustre: lustre-MDT0001: new disk, initializing [ 6663.930743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6663.975948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6663.994125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6670.110663] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6679.488744] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6685.208919] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6692.819957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6693.078427] Lustre: lustre-OST0000: new disk, initializing [ 6693.081857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6693.087230] Lustre: 152374:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6693.170180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6699.276832] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6701.118920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6701.125081] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6701.203902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6707.880506] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6713.584556] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_module obdecho/obdecho [ 6715.160338] LustreError: 153375:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6715.167545] LustreError: 153375:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6716.535954] LustreError: 153472:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6730.730130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6730.738218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6730.749783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6732.257949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6732.268043] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.283258] Lustre: Skipped 1 previous similar message [ 6734.261624] Lustre: server umount lustre-MDT0000 complete [ 6737.521751] LustreError: 150398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575391 with bad export cookie 17583463853152814828 [ 6737.524123] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6737.531714] LustreError: 150398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6737.793649] Lustre: server umount lustre-MDT0001 complete [ 6751.433615] Lustre: server umount lustre-OST0000 complete [ 6777.756539] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 6788.816111] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 6825.270842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6825.558438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6825.583137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6825.641370] Lustre: lustre-MDT0000: new disk, initializing [ 6825.705757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6825.722501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6829.459423] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6839.292501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6839.408941] Lustre: 157189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6839.454923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6839.459277] Lustre: Skipped 1 previous similar message [ 6839.516812] Lustre: lustre-MDT0001: new disk, initializing [ 6839.590667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6839.616180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6839.633925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6843.278794] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6851.885264] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6857.213757] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6863.825145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6864.119622] Lustre: lustre-OST0000: new disk, initializing [ 6864.123866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6864.130349] Lustre: 159157:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6864.227970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6870.886137] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6872.169476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6872.188222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6872.317709] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6881.848477] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6888.075386] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6888.323079] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6893.304572] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6893.491665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6898.150090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6898.153983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6898.167477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6898.177370] Lustre: Skipped 3 previous similar messages [ 6901.965918] Lustre: server umount lustre-OST0000 complete [ 6906.338452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6906.343954] LustreError: Skipped 1 previous similar message [ 6906.347167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6906.356732] Lustre: Skipped 1 previous similar message [ 6906.360387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6912.655530] Lustre: server umount lustre-MDT0000 complete [ 6913.504242] LustreError: 158331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6913.522852] LustreError: 158331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6916.667790] LustreError: 157183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575570 with bad export cookie 17583463853152815787 [ 6916.688969] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6916.689796] LustreError: 157183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6917.114311] Lustre: server umount lustre-MDT0001 complete [ 6920.926915] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 08:46:13 (1787575573) [ 6923.071470] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6925.446316] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 08:46:17 (1787575577) [ 6934.871661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6935.457188] LustreError: 161572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6935.645983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6940.646082] LustreError: 161573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6940.894864] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6945.765521] LustreError: 161572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6950.653424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6955.515483] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6963.099857] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6969.871969] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6977.722567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6978.007932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6978.013905] Lustre: Skipped 1 previous similar message [ 6985.316561] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6995.806861] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7011.815596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7011.822217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7011.848816] Lustre: Skipped 3 previous similar messages [ 7011.873558] Lustre: Skipped 4 previous similar messages [ 7014.560221] Lustre: server umount lustre-MDT0000 complete [ 7016.931032] LustreError: 161572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7016.950578] LustreError: 161572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7018.180423] LustreError: 161552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575672 with bad export cookie 17583463853152816669 [ 7018.182063] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7018.200100] LustreError: 161552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7022.049538] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7022.072524] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7022.087890] Lustre: Skipped 1 previous similar message [ 7024.882747] Lustre: server umount lustre-MDT0001 complete [ 7031.454567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7044.860473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7045.128501] LustreError: 165417:0:(llog.c:1655:llog_backup()) MGC192.168.202.144@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7045.134498] Lustre: 165417:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.144@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7048.673681] LustreError: 165422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7048.693657] LustreError: 165422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7068.128797] LustreError: 165422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.140967] LustreError: 165422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 7103.456247] LustreError: 165422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7103.471393] LustreError: 165422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 7106.018738] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7106.024892] Lustre: 165417:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7147.002185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7151.297864] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7152.100682] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7160.630853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.369904] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf405033698186753 [ 7163.379319] Lustre: MGC192.168.202.144@tcp: Connection restored to 0@lo (at 0@lo) [ 7167.845792] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7169.009462] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7174.503667] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7176.242901] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7181.609138] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7183.574735] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7199.715084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7199.718697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7199.730527] Lustre: Skipped 2 previous similar messages [ 7199.741577] Lustre: Skipped 4 previous similar messages [ 7201.078677] Lustre: server umount lustre-MDT0000 complete [ 7204.834569] LustreError: 165422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7204.855300] LustreError: 165422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 46 previous similar messages [ 7204.962959] LustreError: 161553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575859 with bad export cookie 17583463853152823123 [ 7204.975163] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7204.982228] LustreError: 161553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7205.312423] Lustre: server umount lustre-MDT0001 complete [ 7219.530358] Lustre: server umount lustre-OST0000 complete [ 7242.666209] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 7256.075754] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 7296.933313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7297.061282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7297.082870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7297.134370] Lustre: lustre-MDT0000: new disk, initializing [ 7297.198869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7297.208973] Lustre: Skipped 1 previous similar message [ 7297.219682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7302.984216] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7314.087846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7314.181925] Lustre: 170527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7314.427629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7318.926941] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7328.175886] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7332.967324] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7341.178506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7341.821922] Lustre: lustre-OST0000: new disk, initializing [ 7341.825405] Lustre: Skipped 1 previous similar message [ 7341.836935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7341.849731] Lustre: Skipped 2 previous similar messages [ 7341.859106] Lustre: 172491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7347.338453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7347.347824] Lustre: Skipped 1 previous similar message [ 7347.364392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7347.572052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7349.396177] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7361.606408] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7366.476842] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7366.661133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7371.102375] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7371.385037] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7378.400161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7378.406480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7378.409520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7378.421499] Lustre: Skipped 1 previous similar message [ 7378.439543] Lustre: Skipped 1 previous similar message [ 7379.925876] Lustre: server umount lustre-OST0000 complete [ 7386.080371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7386.089755] LustreError: Skipped 1 previous similar message [ 7389.965780] Lustre: server umount lustre-MDT0000 complete [ 7393.714529] LustreError: 170518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576047 with bad export cookie 17583463853152839951 [ 7393.723973] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7393.724520] LustreError: 170518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7393.998240] Lustre: server umount lustre-MDT0001 complete [ 7402.499219] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:54:14 (1787576054) [ 7408.743659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7409.090509] LustreError: 174906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7413.319786] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7420.398883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7425.478030] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7433.576493] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7439.643360] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7446.090607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7451.866333] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7459.043541] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7468.920397] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:55:21 (1787576121) [ 7475.171876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7475.178715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7475.190053] Lustre: Skipped 3 previous similar messages [ 7475.207403] Lustre: Skipped 4 previous similar messages [ 7479.619673] Lustre: server umount lustre-OST0000 complete [ 7498.207596] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7498.476712] Lustre: server umount lustre-MDT0000 complete [ 7501.980866] LustreError: 175641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576156 with bad export cookie 17583463853152840826 [ 7501.981544] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7502.002354] LustreError: 175641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7502.552604] Lustre: server umount lustre-MDT0001 complete [ 7513.970637] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 7516.770472] Key type lgssc unregistered [ 7517.040631] LNet: 178638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7517.051525] LNetError: 178638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7518.117892] LNet: Removed LNI 192.168.202.144@tcp [ 7519.071456] Key type .llcrypt unregistered [ 7519.073889] Key type ._llcrypt unregistered [ 7539.508075] Key type ._llcrypt registered [ 7539.512100] Key type .llcrypt registered [ 7539.688307] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 7540.916132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7541.023959] alg: No test for adler32 (adler32-zlib) [ 7542.156838] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 7542.572320] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 7544.287188] Key type lgssc registered [ 7545.509085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7553.943858] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7553.961676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7569.504150] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7574.623315] LustreError: 179881:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.144@tcp: failed processing log, type 4: rc = -110 [ 7579.807574] LustreError: 179881:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7579.818966] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7579.826188] LustreError: MGC192.168.202.144@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7579.839911] LustreError: 179881:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7579.856251] LustreError: 179881:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7579.869333] LustreError: 179881:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7579.914297] Lustre: server umount lustre-OST0000 complete [ 7579.916396] LustreError: 179881:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7588.268994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.888188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7594.215096] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7600.097236] LustreError: 180382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7600.124295] LustreError: 180382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7602.363434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7602.673306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7606.802338] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7613.250607] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7619.238989] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7626.067693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7626.359668] LustreError: 182198:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7626.367931] Lustre: 182198:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.144@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7626.529678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7632.989253] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7640.762802] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7643.617698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7643.627264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7643.661253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7646.180119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7646.180997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7646.200245] Lustre: Skipped 1 previous similar message [ 7646.224337] Lustre: Skipped 1 previous similar message [ 7649.085246] Lustre: server umount lustre-MDT0000 complete [ 7651.321980] LustreError: 180377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7651.348522] LustreError: 180377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7653.221675] LustreError: 180363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576307 with bad export cookie 13070076457905090463 [ 7653.237405] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7653.615134] Lustre: server umount lustre-MDT0001 complete [ 7668.701238] Lustre: server umount lustre-OST0000 complete [ 7679.821336] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 7690.199453] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 7731.226328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7731.574507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7731.599268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7731.660841] Lustre: lustre-MDT0000: new disk, initializing [ 7731.759117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7731.771874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7735.824324] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7746.735162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7746.863319] Lustre: 185724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7746.914275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7746.918314] Lustre: Skipped 1 previous similar message [ 7747.006602] Lustre: lustre-MDT0001: new disk, initializing [ 7747.161189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7747.201995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7747.223393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7751.809033] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7761.089245] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7766.951775] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7773.176791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7773.420735] Lustre: lustre-OST0000: new disk, initializing [ 7773.423991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7773.429432] Lustre: 187693:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7773.489798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7779.346781] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7781.980144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7781.985701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7782.070944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7789.245491] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7794.063317] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7794.279933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7799.122944] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7799.305369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7802.848888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7802.859534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7802.885621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7802.898474] Lustre: Skipped 1 previous similar message [ 7807.381620] Lustre: server umount lustre-OST0000 complete [ 7813.088376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.102660] Lustre: Skipped 1 previous similar message [ 7813.110478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7818.209632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7818.212865] Lustre: Skipped 2 previous similar messages [ 7825.375423] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7825.619648] Lustre: server umount lustre-MDT0000 complete [ 7828.453213] LustreError: 185732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7828.481577] LustreError: 185732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7829.258771] LustreError: 185716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576483 with bad export cookie 13070076457905091184 [ 7829.260411] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7829.269100] LustreError: 185716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7829.601735] Lustre: server umount lustre-MDT0001 complete [ 7837.732518] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 09:01:30 (1787576490) [ 7855.391496] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 7858.988563] Key type lgssc unregistered [ 7859.293326] LNet: 190652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7859.309808] LNetError: 190652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7860.393743] LNet: Removed LNI 192.168.202.144@tcp [ 7861.076853] Key type .llcrypt unregistered [ 7861.079471] Key type ._llcrypt unregistered [ 7880.929059] Key type ._llcrypt registered [ 7880.932289] Key type .llcrypt registered [ 7881.052875] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 7882.480699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7882.670475] alg: No test for adler32 (adler32-zlib) [ 7883.848318] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 7884.238440] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 7885.919201] Key type lgssc registered [ 7887.444137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7896.851541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7896.898295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7898.542929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7904.191734] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7907.819021] LustreError: 191884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7907.856291] LustreError: 191884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7912.951058] LustreError: 191885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7913.730454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7914.325430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7919.363823] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7927.423690] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7932.994649] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7941.973404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7942.145749] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7942.148473] LustreError: 193750:0:(llog.c:1696:llog_backup()) MGC192.168.202.144@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7942.154462] Lustre: 193750:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.144@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7942.295845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7949.516777] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7957.433503] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7960.550798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7960.566086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7960.584728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7965.158887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7965.166524] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7970.275639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7970.285241] Lustre: Skipped 3 previous similar messages [ 7974.375104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7974.768964] Lustre: server umount lustre-MDT0000 complete [ 7975.412807] LustreError: 191885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7975.440319] LustreError: 191885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7978.874075] LustreError: 191865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576633 with bad export cookie 10961173385127011874 [ 7978.875834] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7978.883222] LustreError: 191865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7979.688608] Lustre: server umount lustre-MDT0001 complete [ 7993.294578] Lustre: server umount lustre-OST0000 complete [ 8002.871885] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 8013.003276] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 8049.673370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8049.941508] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8049.984138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8050.073797] Lustre: lustre-MDT0000: new disk, initializing [ 8050.184956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8050.203470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8055.292483] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8066.845420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8066.982300] Lustre: 197276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8067.038338] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8067.042652] Lustre: Skipped 1 previous similar message [ 8067.045583] LustreError: 198058:0:(llog.c:1696:llog_backup()) MGC192.168.202.144@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8067.054335] LustreError: 198058:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8067.063271] Lustre: 198058:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.144@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8067.085566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8067.097508] Lustre: Skipped 1 previous similar message [ 8067.265520] Lustre: lustre-MDT0001: new disk, initializing [ 8067.422269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8067.489750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8067.513268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8072.639357] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8082.688244] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8089.283440] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8097.889661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8098.203845] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8098.209574] Lustre: Skipped 1 previous similar message [ 8098.216874] LustreError: 199221:0:(llog.c:1696:llog_backup()) MGC192.168.202.144@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8098.232229] LustreError: 199221:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8098.238529] Lustre: 199221:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.144@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8098.247869] Lustre: 199221:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8098.281857] Lustre: lustre-OST0000: new disk, initializing [ 8098.286343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8098.295932] Lustre: 199244:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8098.417900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8104.528858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8104.543838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8104.667944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8105.178643] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8115.187149] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8120.150295] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8120.386978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8125.618931] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8125.878264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8130.528480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8130.549169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8130.552987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8130.590644] Lustre: Skipped 2 previous similar messages [ 8130.620117] Lustre: Skipped 1 previous similar message [ 8134.571339] Lustre: server umount lustre-OST0000 complete [ 8139.233029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8139.244467] LustreError: Skipped 1 previous similar message [ 8139.253567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8139.275549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8145.473065] Lustre: server umount lustre-MDT0000 complete [ 8145.900185] LustreError: 197282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8145.922716] LustreError: 197282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8149.243654] LustreError: 197267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576803 with bad export cookie 10961173385127012595 [ 8149.248182] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8149.255886] LustreError: 197267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8149.566658] Lustre: server umount lustre-MDT0001 complete [ 8158.737783] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 09:06:51 (1787576811) [ 8176.632903] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 8186.312601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8186.680709] LustreError: 202374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8186.765800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8191.394267] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8196.064443] LustreError: 202375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8196.085482] LustreError: 202375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8200.394661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8205.422398] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8208.799489] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8216.823658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8223.459118] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8231.761583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8231.946640] Lustre: lustre-OST0001: new disk, initializing [ 8231.949635] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8231.957173] Lustre: 204682:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8232.075813] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8232.083446] Lustre: Skipped 2 previous similar messages [ 8237.931090] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8240.798795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8240.818560] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8240.947774] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8248.954878] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8254.956240] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8257.099728] Lustre: DEBUG MARKER: ADD OST3 [ 8261.946706] loop: module loaded [ 8262.023452] loop0: detected capacity change from 0 to 204800000 [ 8262.339540] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8262.355723] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8262.371638] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8267.983317] loop0: detected capacity change from 0 to 204800000 [ 8268.986374] Lustre: DEBUG MARKER: STOP OST3 [ 8273.372454] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8273.563555] Lustre: lustre-OST0003: new disk, initializing [ 8273.570448] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8273.574398] Lustre: 206665:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8275.703854] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8275.908662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8275.922397] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8275.950806] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8277.727278] LustreError: 206863:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8278.334456] Lustre: Failing over lustre-OST0003 [ 8278.544079] Lustre: server umount lustre-OST0003 complete [ 8280.599405] LustreError: 203860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8281.027259] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8281.058743] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8281.070916] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8281.087570] Lustre: Skipped 2 previous similar messages [ 8289.359607] Lustre: DEBUG MARKER: START OST3 again [ 8295.026638] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8295.369963] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8295.933319] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8296.640476] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8296.644208] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8301.433186] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8321.964151] Lustre: Failing over lustre-OST0003 [ 8322.368816] Lustre: server umount lustre-OST0003 complete [ 8322.529123] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8322.541451] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8322.554833] LustreError: 203861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8322.569871] Lustre: Skipped 1 previous similar message [ 8322.615802] LustreError: 203861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 8340.452212] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.458067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8340.466905] Lustre: Skipped 2 previous similar messages [ 8340.480262] Lustre: Skipped 5 previous similar messages [ 8345.933836] Lustre: server umount lustre-MDT0000 complete [ 8349.612058] LustreError: 203110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577003 with bad export cookie 10961173385127013477 [ 8349.614555] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8349.629437] LustreError: 203110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8356.185323] Lustre: server umount lustre-MDT0001 complete [ 8360.865861] Lustre: server umount lustre-OST0000 complete [ 8365.263513] Lustre: server umount lustre-OST0001 complete [ 8373.654881] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 8383.744518] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 8420.155856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8420.508823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8420.552814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8420.750347] Lustre: lustre-MDT0000: new disk, initializing [ 8420.870645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8420.877969] Lustre: Skipped 2 previous similar messages [ 8420.912236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8426.767144] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8437.819275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.998473] Lustre: 212140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8438.098081] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8438.104075] Lustre: Skipped 1 previous similar message [ 8438.195968] Lustre: lustre-MDT0001: new disk, initializing [ 8438.314564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8438.334416] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8442.808382] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8453.162390] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8459.481321] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8467.841911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8468.199497] Lustre: 214112:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8469.859536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8469.993503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8474.954531] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8485.543659] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8489.793749] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8490.074499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8494.265676] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8494.502644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8500.705826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8500.707664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.715887] LustreError: Skipped 2 previous similar messages [ 8500.720702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8500.720711] Lustre: Skipped 8 previous similar messages [ 8500.779793] Lustre: Skipped 4 previous similar messages [ 8503.250746] Lustre: server umount lustre-OST0000 complete [ 8514.064359] Lustre: server umount lustre-MDT0000 complete [ 8516.076696] LustreError: 212147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8516.113869] LustreError: 212147:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 8518.705575] LustreError: 212132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577172 with bad export cookie 10961173385127015773 [ 8518.710820] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8518.718656] LustreError: 212132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8519.103490] Lustre: server umount lustre-MDT0001 complete [ 8523.957419] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 09:12:56 (1787577176) [ 8545.284291] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 8563.706634] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 8606.702722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8607.212641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8607.266349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8607.272587] Lustre: Skipped 1 previous similar message [ 8607.402868] Lustre: lustre-MDT0000: new disk, initializing [ 8607.407843] Lustre: Skipped 1 previous similar message [ 8607.589543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8607.602304] Lustre: Skipped 2 previous similar messages [ 8607.625926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8607.638427] Lustre: Skipped 1 previous similar message [ 8612.946358] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8626.082322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8626.323746] Lustre: 218744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8626.828983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8633.703595] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8646.953247] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8653.178876] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8659.559294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8659.935297] Lustre: 220714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8661.633123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8661.784191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8666.236969] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8674.861308] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8679.583825] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8679.789794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8684.753335] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8685.011593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8687.591129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8687.594543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8687.611103] LustreError: Skipped 1 previous similar message [ 8687.613706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8687.613713] Lustre: Skipped 4 previous similar messages [ 8687.647480] Lustre: Skipped 3 previous similar messages [ 8693.472505] Lustre: server umount lustre-OST0000 complete [ 8703.385123] Lustre: server umount lustre-MDT0000 complete [ 8707.020711] LustreError: 218735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577361 with bad export cookie 10961173385127016655 [ 8707.026783] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8707.038596] LustreError: 218735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8707.522950] Lustre: server umount lustre-MDT0001 complete [ 8716.356947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8716.977959] LustreError: 222929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8724.461559] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8735.272499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8739.826622] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8748.254780] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8753.303047] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8761.488747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8767.271239] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 8774.898586] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8785.381350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8785.386677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8785.391085] Lustre: Skipped 8 previous similar messages [ 8785.402886] LustreError: Skipped 1 previous similar message [ 8786.787385] Lustre: server umount lustre-OST0000 complete [ 8796.913119] Lustre: server umount lustre-MDT0000 complete [ 8801.539601] LustreError: 222911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577455 with bad export cookie 10961173385127017537 [ 8801.557828] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8801.559312] LustreError: 222911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8801.829394] Lustre: server umount lustre-MDT0001 complete [ 8917.867210] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 8922.468345] Key type lgssc unregistered [ 8922.908786] LNet: 226600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8922.913835] LNetError: 226600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8922.927370] LNet: Removed LNI 192.168.202.144@tcp [ 8924.337136] Key type .llcrypt unregistered [ 8924.338352] Key type ._llcrypt unregistered [ 8951.369411] Key type ._llcrypt registered [ 8951.371497] Key type .llcrypt registered [ 8951.507986] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 8967.218366] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 8968.641227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8968.658206] alg: No test for adler32 (adler32-zlib) [ 8969.700176] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 8969.970205] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 8971.751337] Key type lgssc registered [ 8972.914551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9010.122755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9010.155059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9011.607758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9011.662126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9011.820601] Lustre: lustre-MDT0000: new disk, initializing [ 9011.984680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9012.015638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9017.040717] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 9028.061396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9028.171765] Lustre: 229968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9028.217547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9028.220842] Lustre: Skipped 1 previous similar message [ 9028.285554] Lustre: lustre-MDT0001: new disk, initializing [ 9028.366485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9028.419727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9028.439619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9033.818093] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 9043.808528] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9050.672334] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9057.473458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9057.709816] Lustre: lustre-OST0000: new disk, initializing [ 9057.717254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9057.723213] Lustre: 231936:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9057.815381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9064.298681] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 9064.516505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9064.532508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9064.703575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9073.310558] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9079.544787] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9079.972467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9087.512364] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9087.828964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9095.667592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9095.676934] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9095.707298] LustreError: Skipped 1 previous similar message [ 9095.708658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9100.794662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9100.810492] Lustre: Skipped 1 previous similar message [ 9105.375171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9105.972964] Lustre: server umount lustre-OST0000 complete [ 9115.616537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9115.627204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9115.639745] Lustre: Skipped 1 previous similar message [ 9115.645826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9115.657738] Lustre: Skipped 1 previous similar message [ 9116.643540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9118.303750] Lustre: server umount lustre-MDT0000 complete [ 9122.588311] LustreError: 229962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577776 with bad export cookie 6315509753770653204 [ 9122.589718] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9122.601015] LustreError: 229962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9123.177055] Lustre: server umount lustre-MDT0001 complete [ 9147.381476] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 9161.100200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9161.580417] LustreError: 234874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9161.594289] LustreError: 234874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9161.623193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9166.825792] LustreError: 234875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9166.920482] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 9170.917487] LustreError: 234874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9176.037024] LustreError: 234875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9178.980045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9179.717756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9185.551846] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 9189.386486] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9196.960023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9197.425727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9204.002662] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 9212.828081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9212.984963] Lustre: lustre-OST0001: new disk, initializing [ 9212.988928] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9212.996275] Lustre: 237183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9219.732664] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 9221.792769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9221.807556] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9221.880705] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9232.863876] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9241.111817] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9257.563942] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 09:25:08 (1787577908) [ 9261.536282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9261.552103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9261.594650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9261.608648] Lustre: Skipped 2 previous similar messages [ 9268.204032] LustreError: 234870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9268.250505] Lustre: server umount lustre-MDT0000 complete [ 9268.252294] LustreError: 234870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9273.919802] LustreError: 234855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577928 with bad export cookie 6315509753770654086 [ 9273.931656] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9273.935020] LustreError: 234855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9274.807745] Lustre: server umount lustre-MDT0001 complete [ 9288.905892] Lustre: server umount lustre-OST0000 complete [ 9303.810485] Lustre: server umount lustre-OST0001 complete [ 9313.691598] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 9325.290739] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 9367.679449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9367.953710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9367.985699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9368.053312] Lustre: lustre-MDT0000: new disk, initializing [ 9368.175602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9368.192913] Lustre: Skipped 1 previous similar message [ 9368.225034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9373.924948] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 9386.590699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9386.854926] Lustre: 241134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9386.988782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9387.001050] Lustre: Skipped 1 previous similar message [ 9387.116339] Lustre: lustre-MDT0001: new disk, initializing [ 9387.261023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9387.286309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9392.237549] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 9404.700662] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9414.366398] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9426.970216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9427.526910] Lustre: lustre-OST0000: new disk, initializing [ 9427.542109] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9427.560260] Lustre: 243102:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9427.755450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9427.758738] Lustre: Skipped 1 previous similar message [ 9429.587618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9429.607289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9429.851116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9436.838339] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 9448.092731] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9454.452310] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9454.748222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9461.199933] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9461.512958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9465.834321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9465.848954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9465.853372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9465.876539] Lustre: Skipped 4 previous similar messages [ 9465.881614] Lustre: Skipped 4 previous similar messages [ 9470.009010] Lustre: server umount lustre-OST0000 complete [ 9475.222218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9475.260077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9481.927849] Lustre: server umount lustre-MDT0000 complete [ 9485.799069] LustreError: 241140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9485.810340] LustreError: 241140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9487.220546] LustreError: 241125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578141 with bad export cookie 6315509753770656214 [ 9487.236821] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9487.242347] LustreError: 241125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9487.650142] Lustre: server umount lustre-MDT0001 complete [ 9496.227321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9496.724648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9501.978980] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 9502.183043] LustreError: 245323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9502.212527] LustreError: 245323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9512.290238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9519.672752] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 9531.544716] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9539.351981] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9548.268548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9556.684109] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 9566.427991] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9714.668189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9714.685864] Lustre: Skipped 1 previous similar message [ 9714.723283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9714.743628] Lustre: Skipped 4 previous similar messages [ 9719.433923] Lustre: server umount lustre-OST0000 complete [ 9730.020804] LustreError: 245317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9730.023159] Lustre: server umount lustre-MDT0000 complete [ 9730.027569] LustreError: 245317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9734.462646] LustreError: 245302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578388 with bad export cookie 6315509753770657103 [ 9734.465438] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9734.475883] LustreError: 245302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9735.126871] Lustre: server umount lustre-MDT0001 complete [ 9835.544953] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 9839.025161] Key type lgssc unregistered [ 9839.388250] LNet: 248935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9839.398879] LNetError: 248935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9839.411502] LNet: Removed LNI 192.168.202.144@tcp [ 9840.296269] Key type .llcrypt unregistered [ 9840.298402] Key type ._llcrypt unregistered [ 9868.861144] Key type ._llcrypt registered [ 9868.863292] Key type .llcrypt registered [ 9869.005528] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 9892.130953] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 9893.995345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9894.103682] alg: No test for adler32 (adler32-zlib) [ 9895.460604] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 9896.199571] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 9898.191112] Key type lgssc registered [ 9900.466861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9967.327687] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 9979.844593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9979.905171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9981.313475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9981.380600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9981.489316] Lustre: lustre-MDT0000: new disk, initializing [ 9981.597253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9981.615260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9986.680046] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 9998.378123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9998.553752] Lustre: 253030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9998.612877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9998.616974] Lustre: Skipped 1 previous similar message [ 9998.673371] Lustre: lustre-MDT0001: new disk, initializing [ 9998.762319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9998.808805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9998.820700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10004.481939] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10011.045605] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10020.135564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10020.401168] Lustre: lustre-OST0000: new disk, initializing [10020.403416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10020.410664] Lustre: 254663:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10020.549747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10020.978954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10020.990446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10021.136330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10027.753124] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10040.357503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10040.668703] Lustre: lustre-OST0001: new disk, initializing [10040.674414] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10040.688729] Lustre: 255534:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10040.884325] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10048.207937] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10050.740198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10050.767475] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10051.060267] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10061.470291] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10070.616338] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10086.888666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10086.898636] Lustre: Skipped 1 previous similar message [10086.907855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10086.919291] Lustre: Skipped 3 previous similar messages [10092.003000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10092.014191] Lustre: Skipped 1 previous similar message [10092.876912] Lustre: server umount lustre-MDT0000 complete [10098.098980] LustreError: 253825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578752 with bad export cookie 14091297122472068848 [10098.111139] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10098.112965] LustreError: 253825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10098.620529] Lustre: server umount lustre-MDT0001 complete [10112.852456] Lustre: server umount lustre-OST0000 complete [10127.897722] Lustre: server umount lustre-OST0001 complete [10147.261893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10157.310959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10167.688045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10180.061826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10206.291163] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [10218.868407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10218.881123] Lustre: lustre-MDT0000: reset Object Index mappings [10219.283599] LustreError: 260552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10219.291526] LustreError: 260552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10219.382958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10224.552832] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10224.611723] LustreError: 260552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10229.730123] LustreError: 260553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10234.849404] LustreError: 260552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10236.722369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10236.754176] Lustre: lustre-MDT0001: reset Object Index mappings [10237.169085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10243.037302] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10246.776756] Lustre: 261695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10253.665634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10254.128240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10261.166774] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10262.314217] LustreError: 262051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10264.419547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10270.842767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10276.369438] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10278.134524] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10286.516364] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10289.798873] Lustre: 263564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10296.845489] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10296.849622] LustreError: 260548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10296.856492] Lustre: Skipped 2 previous similar messages [10296.878214] LustreError: 260548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10297.961770] Lustre: server umount lustre-MDT0000 complete [10302.519035] LustreError: 260534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578956 with bad export cookie 14091297122472075638 [10302.520878] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10302.528861] LustreError: 260534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10303.409698] Lustre: server umount lustre-MDT0001 complete [10319.738480] Lustre: server umount lustre-OST0000 complete [10323.424542] Lustre: 250514:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787578961/real 1787578961] req@ffff9a21fdd2f100 x1874411965781760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787578977 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10323.487394] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10323.536306] Lustre: Skipped 1 previous similar message [10324.801405] Lustre: server umount lustre-OST0001 complete [10349.019770] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 09:43:21 (1787579001) [10373.724723] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [10389.994099] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [10433.662475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10433.945734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10433.970101] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10434.070859] Lustre: lustre-MDT0000: new disk, initializing [10434.169344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10434.174644] Lustre: Skipped 1 previous similar message [10434.203018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10439.187401] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10452.825604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10452.947748] Lustre: 267842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10452.995255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10452.998807] Lustre: Skipped 1 previous similar message [10453.081978] Lustre: lustre-MDT0001: new disk, initializing [10453.329301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10453.340737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10458.852905] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10470.480569] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10478.219349] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10488.749701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10489.246660] Lustre: lustre-OST0000: new disk, initializing [10489.250616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10489.255886] Lustre: 269810:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10491.317576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10491.329509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10491.543213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10498.401717] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10509.703592] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10514.458807] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10514.680746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10519.581720] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10519.818579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10522.083672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10522.084723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10522.101046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10522.101054] Lustre: Skipped 2 previous similar messages [10522.148329] LustreError: Skipped 1 previous similar message [10527.208804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10527.216070] Lustre: Skipped 2 previous similar messages [10528.253577] Lustre: server umount lustre-OST0000 complete [10535.393796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10535.406947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10535.436423] Lustre: Skipped 1 previous similar message [10535.453418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10538.933988] Lustre: server umount lustre-MDT0000 complete [10542.560147] LustreError: 267854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10542.570277] LustreError: 267854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10543.200162] LustreError: 267836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579197 with bad export cookie 14091297122472077731 [10543.217616] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10543.648337] Lustre: server umount lustre-MDT0001 complete [10567.421049] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [10582.501570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10583.008234] LustreError: 272747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10583.050657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10583.054753] Lustre: Skipped 2 previous similar messages [10587.620994] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10596.992589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10603.235475] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10608.029227] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10615.752456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10623.722210] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10634.199089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10634.441820] Lustre: lustre-OST0001: new disk, initializing [10634.450768] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10634.461923] Lustre: 275056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10636.229805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10636.252117] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10636.382219] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10642.663650] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10654.619567] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10659.979322] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10666.573572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10666.956157] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10677.242504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10677.246404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10677.276490] Lustre: Skipped 4 previous similar messages [10677.294272] Lustre: Skipped 2 previous similar messages [10682.116037] Lustre: server umount lustre-MDT0000 complete [10682.341633] LustreError: 272744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10682.373677] LustreError: 272744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10687.131446] LustreError: 272729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579341 with bad export cookie 14091297122472078613 [10687.133128] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10687.139616] LustreError: 272729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10687.539630] Lustre: server umount lustre-MDT0001 complete [10700.924642] Lustre: server umount lustre-OST0000 complete [10702.816657] Lustre: 250514:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787579341/real 1787579341] req@ffff9a20c2085500 x1874411965897984/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787579357 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10702.852744] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10704.718808] Lustre: server umount lustre-OST0001 complete [10722.907500] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [10734.143978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10734.613336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10734.621849] Lustre: Skipped 3 previous similar messages [10739.190192] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10748.977365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10755.991497] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10759.662505] Lustre: 279255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10768.431196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10778.855485] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10779.402962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10792.182870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10798.152213] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10804.772389] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10815.207417] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10820.261238] Lustre: 281138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10825.794938] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10825.974847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10836.815634] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 09:51:28 (1787579488) [10854.372694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10854.384383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10854.418394] Lustre: Skipped 2 previous similar messages [10854.437981] Lustre: Skipped 6 previous similar messages [10859.001940] Lustre: server umount lustre-MDT0000 complete [10859.501191] LustreError: 280150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10859.530802] LustreError: 280150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [10862.624059] LustreError: 278095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579516 with bad export cookie 14091297122472085207 [10862.649972] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10862.650590] LustreError: 278095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10869.770313] Lustre: server umount lustre-MDT0001 complete [10880.570702] Lustre: server umount lustre-OST0000 complete [10885.054583] Lustre: server umount lustre-OST0001 complete [10892.632354] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [10905.270861] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [10960.651320] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [10975.416924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10975.787722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10975.810750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10975.891699] Lustre: lustre-MDT0000: new disk, initializing [10976.029860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10982.199559] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [10995.752235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10995.960991] Lustre: 285387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10996.529384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10996.550709] Lustre: Skipped 4 previous similar messages [10996.617603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11003.238848] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11009.406114] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11018.670503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11019.165875] Lustre: 287021:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11020.740677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11026.305089] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11041.185963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11041.432667] Lustre: 287894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11042.891671] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11042.920674] Lustre: Skipped 1 previous similar message [11043.055846] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11050.466477] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11063.866038] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11088.539569] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 09:55:40 (1787579740) [11093.974730] Lustre: server umount lustre-MDT0000 complete [11094.004114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11094.033788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11094.071684] Lustre: Skipped 3 previous similar messages [11099.543308] LustreError: 285379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579753 with bad export cookie 14091297122472092858 [11099.574079] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11099.574162] LustreError: 285379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11099.690717] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11099.697886] Lustre: Skipped 4 previous similar messages [11100.447258] Lustre: server umount lustre-MDT0001 complete [11106.427507] Lustre: server umount lustre-OST0000 complete [11111.220230] Lustre: server umount lustre-OST0001 complete [11121.342977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11121.843625] LustreError: 290138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11121.864228] LustreError: 290138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11127.599198] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11138.088290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11144.866193] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11153.615268] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11161.272390] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11169.354060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11177.171296] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11186.997875] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11217.071075] loop1: detected capacity change from 0 to 204800000 [11217.627827] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11217.669085] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11217.700022] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11224.929244] loop1: detected capacity change from 0 to 204800000 [11234.334582] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11234.526747] LustreError: 290128:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs12988: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11234.563753] LustreError: 290128:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs12988 [11234.583422] LustreError: 290128:0:(mgs_handler.c:517:mgs_target_reg()) fs12988-MDT0001: Error checking target index: rc = -13 [11234.614778] LustreError: fs12988-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11234.627810] LustreError: 293862:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11234.636453] LustreError: 293862:0:(tgt_mount.c:2132:server_put_super()) no obd fs12988-MDT0001 [11234.653481] LustreError: 293862:0:(tgt_mount.c:116:server_deregister_mount()) fs12988-MDT0001 not registered [11234.757488] Lustre: server umount fs12988-MDT0001 complete [11234.761851] LustreError: 293862:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11248.169749] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11248.400350] Lustre: Setting parameter fs12988-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs12988-MDT0001 [11248.406825] Lustre: Skipped 1 previous similar message [11248.461807] Lustre: srv-fs12988-MDT0001: No data found on store. Initialize space. [11248.476256] Lustre: Skipped 4 previous similar messages [11248.600860] Lustre: fs12988-MDT0001: new disk, initializing [11248.605849] Lustre: Skipped 3 previous similar messages [11248.831034] Lustre: srv-fs12988-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11248.849330] Lustre: 294332:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12988-MDT0001-mdtlov: can't create new llog: rc = -115 [11248.859568] LustreError: 294332:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12988-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11255.242740] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11263.376065] Lustre: server umount fs12988-MDT0001 complete [11263.977724] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11269.108899] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11280.371837] loop1: detected capacity change from 0 to 204800000 [11287.745528] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11288.840954] Lustre: 295901:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12988-MDT0001-mdtlov: can't create new llog: rc = -115 [11288.861770] LustreError: 295901:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12988-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11295.827344] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11300.816751] Lustre: server umount fs12988-MDT0001 complete [11312.621021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11312.634249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11312.662814] Lustre: Skipped 5 previous similar messages [11312.669860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11317.731146] Lustre: server umount lustre-MDT0000 complete [11326.918655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11327.221403] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11333.126546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11334.599920] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11344.588286] loop1: detected capacity change from 0 to 204800000 [11353.047678] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11353.707269] Lustre: 298347:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12988-MDT0001-mdtlov: can't create new llog: rc = -115 [11353.732782] LustreError: 298347:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12988-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11358.125367] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11364.959899] Lustre: server umount fs12988-MDT0001 complete [11392.995150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11398.690410] Lustre: server umount lustre-OST0000 complete [11409.543877] Lustre: server umount lustre-MDT0000 complete [11413.412803] LustreError: 290118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580067 with bad export cookie 14091297122472095539 [11413.429848] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11413.721915] Lustre: server umount lustre-MDT0001 complete [11427.008393] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [11430.463585] Key type lgssc unregistered [11430.851925] LNet: 300708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11430.857672] LNetError: 300708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11431.913314] LNet: Removed LNI 192.168.202.144@tcp [11433.049288] Key type .llcrypt unregistered [11433.056176] Key type ._llcrypt unregistered [11458.499582] Key type ._llcrypt registered [11458.504242] Key type .llcrypt registered [11458.632839] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [11460.070892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11460.103208] alg: No test for adler32 (adler32-zlib) [11461.314715] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [11461.533744] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [11463.175826] Key type lgssc registered [11464.420561] Lustre: Echo OBD driver; http://www.lustre.org/ [11473.697771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11473.719973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11475.141248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11480.483666] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11485.674580] LustreError: 301980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11485.696807] LustreError: 301980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11490.397350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11490.895384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11496.751218] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11507.080886] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11515.721886] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11523.859185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11524.544617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11531.955239] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11535.852796] LustreError: 303814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11540.994081] LustreError: 303815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11541.796539] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11545.133583] LustreError: 304436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11545.153067] LustreError: 304436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11550.191869] LustreError: 303814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11550.229655] LustreError: 303814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11558.399663] LustreError: 303813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11558.424341] LustreError: 303813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11567.411539] loop1: detected capacity change from 0 to 204800000 [11567.663892] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11567.676435] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11567.701776] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11574.044268] loop1: detected capacity change from 0 to 204800000 [11575.798962] LustreError: 303814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.837330] LustreError: 303814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11579.909613] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11580.213992] Lustre: fs11212-OST0000: new disk, initializing [11580.222895] Lustre: srv-fs11212-OST0000: No data found on store. Initialize space. [11580.234755] Lustre: 305720:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11580.344379] Lustre: fs11212-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11586.247614] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11596.492940] Lustre: server umount fs11212-OST0000 complete [11609.603804] LustreError: 303815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11609.629852] LustreError: 303815:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11613.784902] loop1: detected capacity change from 0 to 204800000 [11616.749675] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11621.873374] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11622.389715] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11622.574076] Lustre: fs11212-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11630.738735] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11642.158121] loop2: detected capacity change from 0 to 204800000 [11642.392818] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11642.414172] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11642.439281] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11650.486545] loop2: detected capacity change from 0 to 204800000 [11658.095046] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11658.217533] LustreError: fs11212-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11658.235452] LustreError: fs11212-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11658.243443] LustreError: 308762:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11658.254355] LustreError: 308762:0:(tgt_mount.c:2132:server_put_super()) no obd fs11212-OST0001 [11658.266639] LustreError: 308762:0:(tgt_mount.c:116:server_deregister_mount()) fs11212-OST0001 not registered [11658.301775] Lustre: server umount fs11212-OST0001 complete [11658.309855] LustreError: 308762:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11671.207190] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11671.430647] Lustre: fs11212-OST0001: new disk, initializing [11671.435072] Lustre: srv-fs11212-OST0001: No data found on store. Initialize space. [11671.442337] Lustre: 309313:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11671.514331] Lustre: fs11212-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11676.165590] LustreError: 304436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11676.192947] LustreError: 304436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11679.134377] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11689.222956] Lustre: server umount fs11212-OST0000 complete [11699.464856] Lustre: server umount fs11212-OST0001 complete [11714.940897] loop1: detected capacity change from 0 to 204800000 [11722.179774] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11722.480394] Lustre: fs11212-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11730.211144] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11738.027401] loop2: detected capacity change from 0 to 204800000 [11746.067232] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11753.792035] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11764.242404] Lustre: server umount fs11212-OST0000 complete [11770.364062] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11775.149949] Lustre: server umount fs11212-OST0001 complete [11775.472455] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11790.837371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11790.843625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11790.858409] Lustre: Skipped 1 previous similar message [11790.859566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11795.079885] Lustre: server umount lustre-OST0000 complete [11801.056464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11801.057324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11801.073704] Lustre: Skipped 1 previous similar message [11801.093566] Lustre: Skipped 2 previous similar messages [11805.586708] Lustre: server umount lustre-MDT0000 complete [11806.180416] LustreError: 301980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11806.203954] LustreError: 301980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 68 previous similar messages [11809.274727] LustreError: 301963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580463 with bad export cookie 8306556343115440613 [11809.278161] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11809.282294] LustreError: 301963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11809.661716] Lustre: server umount lustre-MDT0001 complete [11822.161500] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [11825.222400] Key type lgssc unregistered [11825.478687] LNet: 315243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11825.488802] LNetError: 315243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11825.504264] LNet: Removed LNI 192.168.202.144@tcp [11826.301166] Key type .llcrypt unregistered [11826.302660] Key type ._llcrypt unregistered [11860.750774] Key type ._llcrypt registered [11860.756489] Key type .llcrypt registered [11860.938614] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [11877.800573] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [11879.401359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11879.456143] alg: No test for adler32 (adler32-zlib) [11880.519908] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [11880.850757] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [11882.535511] Key type lgssc registered [11883.854303] Lustre: Echo OBD driver; http://www.lustre.org/ [11920.875827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11920.903596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11922.198744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11922.269183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11922.325977] Lustre: lustre-MDT0000: new disk, initializing [11922.436406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11922.457976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11926.464955] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11937.525928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11937.715469] Lustre: 319109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11937.788969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11937.792888] Lustre: Skipped 1 previous similar message [11937.891431] Lustre: lustre-MDT0001: new disk, initializing [11938.001982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11938.042638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11938.050772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11941.970909] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11951.647972] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11957.488321] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11965.064970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11965.241544] Lustre: lustre-OST0000: new disk, initializing [11965.245777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11965.252463] Lustre: 321074:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11965.355634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11972.271122] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [11973.696079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11973.709885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11973.875793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11981.236488] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11983.017798] Lustre: Mounted lustre-client - version 2.17.57_80_ga77f2be [11983.831782] Lustre: Unmounted lustre-client [11993.583467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11993.604329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11993.605037] Lustre: Skipped 1 previous similar message [11993.628327] Lustre: Skipped 1 previous similar message [11998.728314] Lustre: server umount lustre-OST0000 complete [12003.812479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12003.841264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12003.849668] Lustre: Skipped 1 previous similar message [12008.831147] Lustre: server umount lustre-MDT0000 complete [12013.068097] LustreError: 319903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580667 with bad export cookie 12032786666849259784 [12013.072764] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12013.084270] LustreError: 319903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12013.407493] Lustre: server umount lustre-MDT0001 complete [12023.876140] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [12026.459895] Key type lgssc unregistered [12036.959608] LNet: 323237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12036.967587] LNetError: 323237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12113.764693] LNet: Removed LNI 192.168.202.144@tcp [12114.589348] Key type .llcrypt unregistered [12114.591441] Key type ._llcrypt unregistered [12133.920724] Key type ._llcrypt registered [12133.923744] Key type .llcrypt registered [12134.091295] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [12153.236329] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [12154.796881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12154.986544] alg: No test for adler32 (adler32-zlib) [12156.207818] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [12156.499793] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [12158.239181] Key type lgssc registered [12159.661912] Lustre: Echo OBD driver; http://www.lustre.org/ [12194.191372] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 10:14:06 (1787580846) [12202.248810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12202.273385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12203.696897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12203.770782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12203.845166] Lustre: lustre-MDT0000: new disk, initializing [12203.963840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12203.988517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12209.853374] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12222.396117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12222.555672] Lustre: 326462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12222.697775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12222.703461] Lustre: Skipped 1 previous similar message [12222.833240] Lustre: lustre-MDT0001: new disk, initializing [12222.958051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12223.004606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12223.032772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12228.510530] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12239.453768] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12245.411533] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12254.032629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12254.404223] Lustre: lustre-OST0000: new disk, initializing [12254.407579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12254.414810] Lustre: 328433:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12254.489931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12260.879402] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12264.554313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12264.566316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12264.733884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12270.354479] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12290.533077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12290.533283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12290.533921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12290.557493] Lustre: Skipped 1 previous similar message [12290.562574] LustreError: Skipped 1 previous similar message [12292.980985] Lustre: server umount lustre-OST0000 complete [12299.753195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12299.767724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12299.790611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12299.797271] Lustre: Skipped 1 previous similar message [12300.772952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12300.800519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12300.814310] Lustre: Skipped 1 previous similar message [12303.107550] Lustre: server umount lustre-MDT0000 complete [12306.814285] LustreError: 326454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580961 with bad export cookie 507825553743797155 [12306.819416] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12306.835906] LustreError: 326454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12307.362803] Lustre: server umount lustre-MDT0001 complete [12319.129926] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [12322.522812] Key type lgssc unregistered [12322.858473] LNet: 330342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12322.868057] LNetError: 330342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12323.942637] LNet: Removed LNI 192.168.202.144@tcp [12324.707496] Key type .llcrypt unregistered [12324.709565] Key type ._llcrypt unregistered [12347.442811] Key type ._llcrypt registered [12347.447313] Key type .llcrypt registered [12347.634066] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [12349.067470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12349.281046] alg: No test for adler32 (adler32-zlib) [12350.572350] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [12350.859427] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [12352.511427] Key type lgssc registered [12353.977404] Lustre: Echo OBD driver; http://www.lustre.org/ [12360.946795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12360.970302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12362.487807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12367.273484] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12371.937068] LustreError: 331614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12371.985618] LustreError: 331614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12375.424090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12376.030643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12380.613641] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12389.157792] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12395.927515] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12404.731868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12405.082638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12411.486508] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12418.532462] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12433.281631] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12435.141633] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12440.602933] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.44@tcp (stopping) [12442.106420] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12442.114942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12442.139610] Lustre: Skipped 2 previous similar messages [12442.185550] Lustre: Skipped 2 previous similar messages [12445.701654] LustreError: 331609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12445.852906] Lustre: server umount lustre-MDT0000 complete [12447.204411] LustreError: 332342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12447.233616] LustreError: 332342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12450.012270] LustreError: 331597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581104 with bad export cookie 12169133614790876621 [12450.045080] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12450.831646] LustreError: 331610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12450.865523] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.44@tcp (stopping) [12452.322116] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12455.329170] LustreError: 332348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12455.337336] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12455.371101] LustreError: 332348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12455.412980] Lustre: Skipped 1 previous similar message [12456.580875] Lustre: server umount lustre-MDT0001 complete [12461.815391] Lustre: server umount lustre-OST0000 complete [12474.108226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12480.117415] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12484.211498] Lustre: 335795:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.144@tcp [12486.814756] Lustre: server umount MGS complete [12497.637699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12497.993764] LustreError: 336368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12498.012942] LustreError: 336368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12498.083106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12503.542645] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12513.939173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12514.286181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12519.910454] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12530.092803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12530.538552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12538.917945] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12547.059805] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12548.675279] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12555.258236] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12556.800832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12565.472138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12565.485397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12565.508453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12565.513022] Lustre: Skipped 1 previous similar message [12568.567122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12568.585509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12569.965422] Lustre: server umount lustre-OST0000 complete [12575.715590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12575.731827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12575.747044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12575.750317] Lustre: Skipped 2 previous similar messages [12580.463843] Lustre: server umount lustre-MDT0000 complete [12583.905618] LustreError: 338481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12583.924702] LustreError: 338481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [12584.032304] LustreError: 336348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581238 with bad export cookie 12169133614790877461 [12584.038407] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12584.045903] LustreError: 336348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12584.611796] Lustre: server umount lustre-MDT0001 complete [12596.424215] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [12599.298241] Key type lgssc unregistered [12599.604051] LNet: 339809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12599.610414] LNetError: 339809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12599.636149] LNet: Removed LNI 192.168.202.144@tcp [12600.504965] Key type .llcrypt unregistered [12600.506946] Key type ._llcrypt unregistered [12621.053528] Key type ._llcrypt registered [12621.056798] Key type .llcrypt registered [12621.168918] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [12638.411323] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [12639.802853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12640.114409] alg: No test for adler32 (adler32-zlib) [12641.412661] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [12641.708430] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [12643.479941] Key type lgssc registered [12644.838979] Lustre: Echo OBD driver; http://www.lustre.org/ [12682.067809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12682.147326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12683.338539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12683.366297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12683.463149] Lustre: lustre-MDT0000: new disk, initializing [12683.586657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12683.602909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12688.441963] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12700.793481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12700.978251] Lustre: 342969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12701.037719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12701.042706] Lustre: Skipped 1 previous similar message [12701.225222] Lustre: lustre-MDT0001: new disk, initializing [12701.387467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12701.457749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12701.474786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12706.840930] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12717.165181] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12725.064963] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12733.144524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12733.701237] Lustre: lustre-OST0000: new disk, initializing [12733.705799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12733.711688] Lustre: 344940:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12733.834586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12740.283743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12740.292671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12740.491772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12742.738694] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12756.302397] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12761.937768] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12762.106517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12767.509944] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12767.758102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12771.297126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12771.308046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12771.330310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12771.355316] Lustre: Skipped 1 previous similar message [12776.419285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12776.428865] Lustre: Skipped 1 previous similar message [12776.735688] Lustre: server umount lustre-OST0000 complete [12783.584520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12783.592555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12783.620419] Lustre: Skipped 1 previous similar message [12783.642594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12786.663161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12786.666477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12786.713580] Lustre: Skipped 1 previous similar message [12788.885175] Lustre: server umount lustre-MDT0000 complete [12793.392953] LustreError: 342961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581447 with bad export cookie 14561505294369559692 [12793.400672] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12793.412878] LustreError: 342961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12793.901884] Lustre: server umount lustre-MDT0001 complete [12804.947596] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 10:24:17 (1787581457) [12824.966300] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [12837.100663] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [12872.136590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12872.461230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12872.484473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12872.565544] Lustre: lustre-MDT0000: new disk, initializing [12872.669991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12872.693319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12877.282499] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12890.506580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12890.747241] Lustre: 349703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12890.895747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12890.911030] Lustre: Skipped 1 previous similar message [12891.177249] Lustre: lustre-MDT0001: new disk, initializing [12891.469898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12891.576658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12891.607529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12898.999600] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12909.458503] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12915.483538] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12923.626063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12924.000461] Lustre: lustre-OST0000: new disk, initializing [12924.004625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12924.012328] Lustre: 351672:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12924.129578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12929.662644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12929.676049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12929.841361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12933.461274] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [12944.843600] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12981.216966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12981.223533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12981.240763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12981.256532] Lustre: Skipped 1 previous similar message [12987.281662] Lustre: server umount lustre-OST0000 complete [12994.017279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12994.027883] LustreError: Skipped 1 previous similar message [12994.035818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12994.051182] Lustre: Skipped 1 previous similar message [12994.060469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12994.067479] Lustre: Skipped 2 previous similar messages [12998.007784] Lustre: server umount lustre-MDT0000 complete [13001.699636] LustreError: 349714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13001.713849] LustreError: 349714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13002.302453] LustreError: 352810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581656 with bad export cookie 14561505294369560567 [13002.317203] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13002.318811] LustreError: 352810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13002.646944] Lustre: server umount lustre-MDT0001 complete [13014.730636] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [13017.939565] Key type lgssc unregistered [13018.368386] LNet: 353852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13018.397560] LNetError: 353852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13019.440536] LNet: Removed LNI 192.168.202.144@tcp [13020.344109] Key type .llcrypt unregistered [13020.351878] Key type ._llcrypt unregistered [13118.188363] Key type ._llcrypt registered [13118.191316] Key type .llcrypt registered [13118.322427] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [13118.643800] Key type .llcrypt unregistered [13118.645319] Key type ._llcrypt unregistered [13207.823321] Key type ._llcrypt registered [13207.828670] Key type .llcrypt registered [13208.056350] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [13208.668569] Key type .llcrypt unregistered [13208.670344] Key type ._llcrypt unregistered [13245.713293] Key type ._llcrypt registered [13245.714503] Key type .llcrypt registered [13245.802374] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [13246.051466] Key type .llcrypt unregistered [13246.054348] Key type ._llcrypt unregistered [13274.397235] Key type ._llcrypt registered [13274.400169] Key type .llcrypt registered [13274.486435] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [13274.797522] Key type .llcrypt unregistered [13274.801123] Key type ._llcrypt unregistered [13354.616454] Key type ._llcrypt registered [13354.619209] Key type .llcrypt registered [13354.764412] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [13355.262539] Key type .llcrypt unregistered [13355.268050] Key type ._llcrypt unregistered [13388.399968] Key type ._llcrypt registered [13388.402646] Key type .llcrypt registered [13388.493592] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [13388.767731] Key type .llcrypt unregistered [13388.773715] Key type ._llcrypt unregistered [13423.074398] Key type ._llcrypt registered [13423.075944] Key type .llcrypt registered [13423.152350] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [13423.440636] Key type .llcrypt unregistered [13423.445859] Key type ._llcrypt unregistered [13454.074590] Key type ._llcrypt registered [13454.077316] Key type .llcrypt registered [13454.176872] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [13454.457393] Key type .llcrypt unregistered [13454.460936] Key type ._llcrypt unregistered [13487.543604] Key type ._llcrypt registered [13487.545750] Key type .llcrypt registered [13487.665488] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [13487.908235] Key type .llcrypt unregistered [13487.910861] Key type ._llcrypt unregistered [13525.010766] Key type ._llcrypt registered [13525.012895] Key type .llcrypt registered [13525.082806] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [13525.270516] Key type .llcrypt unregistered [13525.272649] Key type ._llcrypt unregistered [13591.796363] Key type ._llcrypt registered [13591.798571] Key type .llcrypt registered [13591.904654] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [13592.329629] Key type .llcrypt unregistered [13592.337798] Key type ._llcrypt unregistered [13622.833484] Key type ._llcrypt registered [13622.835386] Key type .llcrypt registered [13622.978314] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [13623.352312] Key type .llcrypt unregistered [13623.359334] Key type ._llcrypt unregistered [13657.930994] Key type ._llcrypt registered [13657.933533] Key type .llcrypt registered [13658.061674] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [13658.347820] Key type .llcrypt unregistered [13658.351546] Key type ._llcrypt unregistered [13714.824609] Key type ._llcrypt registered [13714.827110] Key type .llcrypt registered [13714.969457] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [13732.034685] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [13732.878171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13733.050171] alg: No test for adler32 (adler32-zlib) [13734.021596] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [13734.205216] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [13735.831184] Key type lgssc registered [13737.113656] Lustre: Echo OBD driver; http://www.lustre.org/ [13775.913916] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13775.938836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13777.140469] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13777.165323] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13777.290810] Lustre: tst250fs-MDT0000: new disk, initializing [13777.368203] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13777.381197] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13781.616687] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [13791.801532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13791.920571] Lustre: 374205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13791.973870] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13791.979462] Lustre: Skipped 1 previous similar message [13792.043894] Lustre: tst250fs-MDT0001: new disk, initializing [13792.126169] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13792.170725] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13792.187748] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13797.069517] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [13802.165559] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13808.648713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13808.863460] Lustre: tst250fs-OST0000: new disk, initializing [13808.866966] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13808.872496] Lustre: 375835:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13808.950659] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13809.430268] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13809.444374] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13809.603445] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13815.935938] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [13827.766450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13827.961210] Lustre: tst250fs-OST0001: new disk, initializing [13827.967977] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13827.975842] Lustre: 376709:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13828.101226] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13834.926592] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13834.949332] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13835.115259] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13835.421574] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [13841.338739] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13844.129199] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13844.135819] Lustre: Skipped 1 previous similar message [13865.954618] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13865.972333] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13865.989138] Lustre: Skipped 1 previous similar message [13866.013547] Lustre: Skipped 3 previous similar messages [13870.446969] Lustre: server umount tst250fs-MDT0000 complete [13874.941921] LustreError: 374198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582529 with bad export cookie 4841244433410979271 [13874.948044] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13874.957264] LustreError: 374198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13875.549495] Lustre: server umount tst250fs-MDT0001 complete [13890.702107] Lustre: server umount tst250fs-OST0000 complete [13892.483455] Lustre: 372432:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787582530/real 1787582530] req@ffff9a21f4a95180 x1874415990151552/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787582546 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13892.527834] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13892.563135] Lustre: Skipped 2 previous similar messages [13895.650755] Lustre: 372429:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787582533/real 1787582533] req@ffff9a20c32fbb80 x1874415990151808/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787582549 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13896.574908] Lustre: server umount tst250fs-OST0001 complete [13919.723550] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [13931.227208] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [13966.697967] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 10:43:38 (1787582618) [13968.319214] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13988.439512] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [14000.655985] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [14035.655787] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13768 sec ========= 10:44:47 (1787582687) [14037.446436] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:44:49 (1787582689) === [14041.591905] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:44:53 (1787582693) === [14083.616698] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [14087.039818] Key type lgssc unregistered [14087.449680] LNet: 385863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14087.457688] LNetError: 385863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14087.481494] LNet: Removed LNI 192.168.202.144@tcp [14088.621176] Key type .llcrypt unregistered [14088.623053] Key type ._llcrypt unregistered