[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 811694202 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.571428] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.574357] pid_max: default: 32768 minimum: 301 [ 0.575976] Security Framework initialized [ 0.577361] SELinux: Initializing. [ 0.578537] SELinux: Starting in permissive mode [ 0.580225] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.588176] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.591511] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.594064] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.598097] Initializing cgroup subsys memory [ 0.599623] Initializing cgroup subsys devices [ 0.601740] Initializing cgroup subsys freezer [ 0.603282] Initializing cgroup subsys net_cls [ 0.606003] Initializing cgroup subsys blkio [ 0.608969] Initializing cgroup subsys perf_event [ 0.611654] Initializing cgroup subsys hugetlb [ 0.613218] Initializing cgroup subsys pids [ 0.615176] Initializing cgroup subsys net_prio [ 0.616910] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.619598] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.621021] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.622386] tlb_flushall_shift: 6 [ 0.623116] FEATURE SPEC_CTRL Present [ 0.623757] FEATURE IBPB_SUPPORT Present [ 0.624419] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.625531] Spectre V2 : Vulnerable [ 0.626275] Speculative Store Bypass: Vulnerable [ 0.627993] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.636475] ACPI: Core revision 20130517 [ 0.639880] ACPI: All ACPI Tables successfully acquired [ 0.641439] ftrace: allocating 30294 entries in 119 pages [ 0.707023] Enabling x2apic [ 0.708054] Enabled x2apic [ 0.709552] Switched APIC routing to physical x2apic. [ 0.713510] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.715756] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.719131] TSC deadline timer enabled [ 0.719305] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.721644] ... version: 2 [ 0.722798] ... bit width: 48 [ 0.723834] ... generic registers: 4 [ 0.724930] ... value mask: 0000ffffffffffff [ 0.726290] ... max period: 00007fffffffffff [ 0.727678] ... fixed-purpose events: 3 [ 0.728919] ... event mask: 000000070000000f [ 0.730649] KVM setup paravirtual spinlock [ 0.735835] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.746795] KVM setup async PF for cpu 1 [ 0.748099] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.761415] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.734156] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.776293] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.783122] KVM setup async PF for cpu 2 [ 0.784268] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.785572] Brought up 4 CPUs [ 0.786645] smpboot: Max logical packages: 1 [ 0.788070] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.791509] KVM setup async PF for cpu 3 [ 0.796674] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.811304] devtmpfs: initialized [ 0.815488] x86/mm: Memory block size: 128MB [ 0.824329] EVM: security.selinux [ 0.828160] EVM: security.ima [ 0.829105] EVM: security.capability [ 0.837196] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.847951] NET: Registered protocol family 16 [ 0.850085] cpuidle: using governor haltpoll [ 0.852981] ACPI: bus type PCI registered [ 0.854506] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.857471] PCI: Using configuration type 1 for base access [ 0.860527] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.874803] ACPI: Added _OSI(Module Device) [ 0.875925] ACPI: Added _OSI(Processor Device) [ 0.877004] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.878151] ACPI: Added _OSI(Processor Aggregator Device) [ 0.879703] ACPI: Added _OSI(Linux-Dell-Video) [ 0.884979] ACPI: EC: Look up EC in DSDT [ 0.888073] ACPI: Interpreter enabled [ 0.889877] ACPI: (supports S0 S3 S4 S5) [ 0.891605] ACPI: Using IOAPIC for interrupt routing [ 0.895066] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.897695] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.905509] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.910549] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.913075] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.918189] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.922436] acpiphp: Slot [2] registered [ 0.924336] acpiphp: Slot [3] registered [ 0.926041] acpiphp: Slot [4] registered [ 0.927341] acpiphp: Slot [5] registered [ 0.929509] acpiphp: Slot [6] registered [ 0.936473] acpiphp: Slot [7] registered [ 0.938823] acpiphp: Slot [8] registered [ 0.944144] acpiphp: Slot [9] registered [ 0.945332] acpiphp: Slot [10] registered [ 0.946594] acpiphp: Slot [11] registered [ 0.960831] acpiphp: Slot [12] registered [ 0.962227] acpiphp: Slot [13] registered [ 0.968016] acpiphp: Slot [14] registered [ 0.971823] acpiphp: Slot [15] registered [ 0.975119] acpiphp: Slot [16] registered [ 0.978783] acpiphp: Slot [17] registered [ 0.983403] acpiphp: Slot [18] registered [ 0.984885] acpiphp: Slot [19] registered [ 0.987631] acpiphp: Slot [20] registered [ 0.989254] acpiphp: Slot [21] registered [ 0.990476] acpiphp: Slot [22] registered [ 0.992290] acpiphp: Slot [23] registered [ 0.995580] acpiphp: Slot [24] registered [ 0.997352] acpiphp: Slot [25] registered [ 0.998837] acpiphp: Slot [26] registered [ 1.001096] acpiphp: Slot [27] registered [ 1.003174] acpiphp: Slot [28] registered [ 1.005933] acpiphp: Slot [29] registered [ 1.009565] acpiphp: Slot [30] registered [ 1.011731] acpiphp: Slot [31] registered [ 1.014398] PCI host bridge to bus 0000:00 [ 1.017728] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 1.020782] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 1.024533] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 1.027510] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 1.032415] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 1.035112] pci_bus 0000:00: root bus resource [bus 00-ff] [ 1.037484] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 1.038535] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 1.039488] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 1.053207] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 1.056652] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 1.060792] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 1.065292] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 1.069125] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 1.074298] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 1.075107] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 1.078616] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 1.081721] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 1.085131] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 1.095615] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 1.100300] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 1.101870] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 1.110614] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 1.114769] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 1.149303] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 1.161798] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.171071] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.182007] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.232826] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 1.243860] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.258206] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.271172] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.296948] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 1.307534] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.317069] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.330378] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.351541] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 1.359793] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.367903] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.374360] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.395751] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 1.400881] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.410688] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.421532] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.437176] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 1.445950] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.453585] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.455418] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.457300] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.459534] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.464186] vgaarb: loaded [ 1.466455] SCSI subsystem initialized [ 1.469133] ACPI: bus type USB registered [ 1.471867] usbcore: registered new interface driver usbfs [ 1.473999] usbcore: registered new interface driver hub [ 1.476545] usbcore: registered new device driver usb [ 1.479860] PCI: Using ACPI for IRQ routing [ 1.481551] PCI: pci_cache_line_size set to 64 bytes [ 1.481959] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.481965] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.481968] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.482301] NetLabel: Initializing [ 1.483516] NetLabel: domain hash size = 128 [ 1.484920] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.486641] NetLabel: unlabeled traffic allowed by default [ 1.489039] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.491292] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.497242] amd_nb: Cannot enumerate AMD northbridges [ 1.502413] Switched to clocksource kvm-clock [ 1.535633] pnp: PnP ACPI init [ 1.539669] ACPI: bus type PNP registered [ 1.543129] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.543217] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.543303] pnp 00:02: [dma 2] [ 1.543346] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.543437] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.545316] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.545411] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.545878] pnp: PnP ACPI: found 6 devices [ 1.551062] ACPI: bus type PNP unregistered [ 1.586134] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.586141] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.586144] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.586147] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.586149] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.586230] NET: Registered protocol family 2 [ 1.593586] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.596474] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.600204] TCP: Hash tables configured (established 32768 bind 32768) [ 1.601977] TCP: reno registered [ 1.603192] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.606237] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.609591] NET: Registered protocol family 1 [ 1.613688] RPC: Registered named UNIX socket transport module. [ 1.616010] RPC: Registered udp transport module. [ 1.617376] RPC: Registered tcp transport module. [ 1.618827] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.621329] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.623348] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.625252] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.627176] PCI: CLS 0 bytes, default 64 [ 1.632932] Unpacking initramfs... [ 4.041435] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 4.044849] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.047416] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 4.053130] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 4.055767] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 4.057555] RAPL PMU: hw unit of domain package 2^-0 Joules [ 4.059195] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 4.066865] cryptomgr_test (52) used greatest stack depth: 14480 bytes left [ 4.070171] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 4.072813] Initialise system trusted keyring [ 4.110826] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 4.112728] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 4.122644] zpool: loaded [ 4.123625] zbud: loaded [ 4.124796] VFS: Disk quotas dquot_6.6.0 [ 4.126244] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 4.132374] NFS: Registering the id_resolver key type [ 4.133711] Key type id_resolver registered [ 4.134763] Key type id_legacy registered [ 4.136017] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 4.138520] Key type big_key registered [ 4.139680] SELinux: Registering netfilter hooks [ 4.145018] cryptomgr_test (58) used greatest stack depth: 14048 bytes left [ 4.147605] cryptomgr_test (63) used greatest stack depth: 13984 bytes left [ 4.148801] NET: Registered protocol family 38 [ 4.148812] Key type asymmetric registered [ 4.148815] Asymmetric key parser 'x509' registered [ 4.148933] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 4.150147] io scheduler noop registered [ 4.150151] io scheduler deadline registered (default) [ 4.150217] io scheduler cfq registered [ 4.150221] io scheduler mq-deadline registered [ 4.150225] io scheduler kyber registered [ 4.152754] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.152763] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.177147] intel_idle: does not run on family 6 model 62 [ 4.177313] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.179505] ACPI: Power Button [PWRF] [ 4.181106] GHES: HEST is not enabled! [ 4.249207] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 4.316873] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 4.554755] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 4.629669] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 4.789726] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.826230] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 4.856342] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 4.859867] Non-volatile memory driver v1.3 [ 4.861450] Linux agpgart interface v0.103 [ 4.863314] crash memory driver: version 1.1 [ 4.865406] nbd: registered device at major 43 [ 4.873676] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 4.873721] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 4.873763] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 4.873819] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 4.873860] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 4.882259] virtio_blk virtio1: [vda] 67296 512-byte logical blocks (34.4 MB/32.8 MiB) [ 4.903713] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 4.903750] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 4.903787] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 4.903817] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 4.903847] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 4.915994] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.929065] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 4.929122] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 4.929167] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 4.929212] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 4.929249] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 4.941247] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.948073] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 4.948108] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 4.948143] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 4.948183] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 4.948214] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 4.961295] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.971100] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 4.971148] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 4.971197] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 4.971235] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 4.971923] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 4.991855] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.010740] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 5.010786] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 5.010834] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 5.010873] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 5.010910] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 5.026657] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.047308] rdac: device handler registered [ 5.051678] hp_sw: device handler registered [ 5.054658] emc: device handler registered [ 5.056490] libphy: Fixed MDIO Bus: probed [ 5.064316] tsc: Refined TSC clocksource calibration: 2399.977 MHz [ 5.068247] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 5.068276] ehci-pci: EHCI PCI platform driver [ 5.068319] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 5.068326] ohci-pci: OHCI PCI platform driver [ 5.068358] uhci_hcd: USB Universal Host Controller Interface driver [ 5.068493] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.073394] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.073437] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.075640] mousedev: PS/2 mouse device common for all mice [ 5.079276] rtc_cmos 00:05: RTC can wake from S4 [ 5.079644] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 5.079873] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.105210] hidraw: raw HID events driver (C) Jiri Kosina [ 5.105495] usbcore: registered new interface driver usbhid [ 5.105495] usbhid: USB HID core driver [ 5.105577] drop_monitor: Initializing network drop monitor service [ 5.105632] Netfilter messages via NETLINK v0.30. [ 5.105705] TCP: cubic registered [ 5.105712] Initializing XFRM netlink socket [ 5.106019] NET: Registered protocol family 10 [ 5.117478] NET: Registered protocol family 17 [ 5.117531] Key type dns_resolver registered [ 5.117928] mce: Using 10 MCE banks [ 5.118174] Loading compiled-in X.509 certificates [ 5.126508] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 5.126620] registered taskstats version 1 [ 5.131243] modprobe (72) used greatest stack depth: 13456 bytes left [ 5.145720] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.151149] Key type trusted registered [ 5.174065] Key type encrypted registered [ 5.174127] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 5.176173] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 5.192069] rtc_cmos 00:05: setting system clock to 2024-04-10 14:56:38 UTC (1712760998) [ 5.247708] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 5.251200] Write protecting the kernel read-only data: 12288k [ 5.256397] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 5.259904] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 5.270185] random: systemd: uninitialized urandom read (16 bytes read) [ 5.276695] random: systemd: uninitialized urandom read (16 bytes read) [ 5.281070] random: systemd: uninitialized urandom read (16 bytes read) [ 5.287009] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 5.297465] systemd[1]: Detected virtualization kvm. [ 5.299197] systemd[1]: Detected architecture x86-64. [ 5.301829] systemd[1]: Running in initial RAM disk. [ 5.315641] systemd[1]: No hostname configured. [ 5.319693] systemd[1]: Set hostname to . [ 5.321531] random: systemd: uninitialized urandom read (16 bytes read) [ 5.324015] systemd[1]: Initializing machine ID from random generator. [ 5.422630] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 5.426316] random: systemd: uninitialized urandom read (16 bytes read) [ 5.428574] random: systemd: uninitialized urandom read (16 bytes read) [ 5.430557] random: systemd: uninitialized urandom read (16 bytes read) [ 5.432456] random: systemd: uninitialized urandom read (16 bytes read) [ 5.435786] random: systemd: uninitialized urandom read (16 bytes read) [ 5.443051] random: systemd: uninitialized urandom read (16 bytes read) [ 5.452886] systemd[1]: Reached target Local File Systems. [ 5.456826] systemd[1]: Reached target Timers. [ 5.460684] systemd[1]: Created slice Root Slice. [ 5.464365] systemd[1]: Created slice System Slice. [ 5.467848] systemd[1]: Listening on udev Control Socket. [ 5.473637] systemd[1]: Listening on Journal Socket. [ 5.490875] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 5.529852] systemd[1]: Starting Journal Service... [ 5.536336] systemd[1]: Starting Load Kernel Modules... [ 5.549343] systemd[1]: Reached target Slices. [ 5.558790] systemd[1]: Starting Setup Virtual Console... [ 5.570149] systemd[1]: Listening on udev Kernel Socket. [ 5.587117] systemd[1]: Reached target Sockets. [ 5.601376] systemd[1]: Starting dracut cmdline hook... [ 5.616756] systemd[1]: Reached target Swap. [ 5.642076] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 5.665721] systemd[1]: Started Load Kernel Modules. [ 5.677522] systemd[1]: Started Setup Virtual Console. [ 5.690116] systemd[1]: Started Journal Service. [ 6.106217] random: fast init done [ 6.136901] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 6.925286] libata version 3.00 loaded. [ 6.989057] ata_piix 0000:00:01.1: version 2.13 [ 7.001959] scsi host0: ata_piix [ 7.007068] scsi host1: ata_piix [ 7.029560] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.039158] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 7.637235] ip (321) used greatest stack depth: 13080 bytes left [ 7.748187] ip (344) used greatest stack depth: 12336 bytes left [ 11.626381] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 12.278023] systemd-journald[101]: Received SIGTERM from PID 1 (systemd). [ 12.802554] SELinux: Disabled at runtime. [ 12.810683] SELinux: Unregistering netfilter hooks [ 12.995872] ip_tables: (C) 2000-2006 Netfilter Core Team [ 13.008896] systemd[1]: Inserted module 'ip_tables' [ 14.544099] systemd-journald[569]: Received request to flush runtime journal from PID 1 [ 15.165826] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 15.303078] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.427769] cryptd: max_cpu_qlen set to 1000 [ 15.679835] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 15.743603] AVX version of gcm_enc/dec engaged. [ 15.748605] AES CTR mode by8 optimization enabled [ 15.783829] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 15.817996] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 15.820058] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 16.146084] cpio (701) used greatest stack depth: 11824 bytes left [ 16.651082] EDAC MC: Ver: 3.0.0 [ 16.694095] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 16.694115] EDAC sbridge: Ver: 1.1.2 [ 24.124835] mount.nfs (779) used greatest stack depth: 10704 bytes left [ 37.303030] device-mapper: uevent: version 1.0.3 [ 37.306850] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 45.067674] libcfs: loading out-of-tree module taints kernel. [ 45.070730] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 45.115040] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 51.480173] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 51.772672] alg: No test for adler32 (adler32-zlib) [ 52.524346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.723438] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 52.952717] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 52.955039] LNet: Accept secure, port 988 [ 54.529428] Key type lgssc registered [ 54.953598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 59.149122] icp: module license 'CDDL' taints kernel. [ 59.150568] Disabling lock debugging due to kernel taint [ 62.226341] ZFS: Loaded module v0.8.6-1, ZFS pool version 5000, ZFS filesystem version 5 [ 66.242357] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 72.394414] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.350897] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 75.360499] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.035986] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 79.046069] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.925083] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 89.556847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.580580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 89.595515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 90.693714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 90.706526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 90.750860] Lustre: lustre-MDT0000: new disk, initializing [ 90.800095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 90.808556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 90.865810] mount.lustre (6549) used greatest stack depth: 9648 bytes left [ 91.947824] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 96.930103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.958550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 96.989988] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 96.999300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 97.001108] Lustre: Skipped 1 previous similar message [ 97.045622] Lustre: lustre-MDT0001: new disk, initializing [ 97.078556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 97.093861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 97.101040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 98.137958] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 104.164120] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 104.170549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 104.199609] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 104.209232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 104.292651] Lustre: lustre-OST0000: new disk, initializing [ 104.295322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 104.325151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 105.936863] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.591487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 107.595732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 107.609529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 110.990212] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 110.998588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.036536] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 111.044345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 111.089230] Lustre: lustre-OST0001: new disk, initializing [ 111.091617] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 111.118279] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 113.051011] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 116.427288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 116.432692] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 116.457211] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 118.960151] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 123.752974] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 129.580899] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing check_logdir /tmp/testlogs/ [ 130.579991] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing yml_node [ 130.849722] random: crng init done [ 131.966154] Lustre: DEBUG MARKER: Client: 2.15.62.1 [ 132.768792] Lustre: DEBUG MARKER: MDS: 2.15.62.1 [ 133.623156] Lustre: DEBUG MARKER: OSS: 2.15.62.1 [ 134.192916] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Apr 10 10:58:46 EDT 2024 [ 136.089830] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 136.551380] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 151.494237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 151.494792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 151.503997] Lustre: Skipped 2 previous similar messages [ 153.486485] Lustre: server umount lustre-MDT0000 complete [ 154.837275] LustreError: 6557:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761148 with bad export cookie 3086013104066358165 [ 154.837954] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 154.847345] LustreError: 6557:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 156.503462] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 156.503863] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 156.505144] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 156.505145] Lustre: Skipped 3 previous similar messages [ 156.527302] Lustre: Skipped 2 previous similar messages [ 159.509927] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 159.509929] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 159.509932] LustreError: Skipped 1 previous similar message [ 159.522524] Lustre: Skipped 2 previous similar messages [ 161.001016] Lustre: server umount lustre-MDT0001 complete [ 168.346093] Lustre: server umount lustre-OST0000 complete [ 175.747831] Lustre: server umount lustre-OST0001 complete [ 177.907828] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 180.548513] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 184.083770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 186.777563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 189.126561] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 189.134183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 191.926853] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 191.940619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 195.087422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 195.126054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.246104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 195.262956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 195.301176] Lustre: lustre-MDT0000: new disk, initializing [ 195.402587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.419644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 196.917943] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 202.451439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.497421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 202.536382] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 202.657193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 202.697626] mount.lustre (16896) used greatest stack depth: 9552 bytes left [ 204.130513] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 207.338943] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 208.384706] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 211.185787] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 211.192918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 211.224144] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 211.232117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 211.364063] Lustre: lustre-OST0000: new disk, initializing [ 211.367744] Lustre: Skipped 1 previous similar message [ 211.371962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 211.375642] Lustre: Skipped 2 previous similar messages [ 211.419981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.424138] Lustre: Skipped 1 previous similar message [ 212.674477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 212.678258] Lustre: Skipped 1 previous similar message [ 212.681473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.719592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 213.324105] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 216.220658] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 217.589197] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 217.698002] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 218.988828] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 219.074452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 222.710629] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 222.717854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 222.726270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 225.910908] Lustre: server umount lustre-OST0000 complete [ 227.670963] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 227.674783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 227.681039] Lustre: Skipped 1 previous similar message [ 227.688129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 227.692012] Lustre: Skipped 1 previous similar message [ 233.357031] Lustre: server umount lustre-MDT0000 complete [ 234.662806] LustreError: 15956:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761227 with bad export cookie 3086013104066359607 [ 234.666997] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 234.671821] LustreError: 15956:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 234.826000] Lustre: server umount lustre-MDT0001 complete [ 237.979513] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 11:00:30 (1712761230) [ 240.560177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 240.813087] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 240.825846] LustreError: Skipped 1 previous similar message [ 240.862726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.114985] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 245.224173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 246.587201] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 248.081110] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 248.895210] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 251.232553] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 251.239148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 253.260868] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 254.857915] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 260.443000] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 265.399109] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 265.403344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.409353] Lustre: Skipped 1 previous similar message [ 265.411581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.414200] Lustre: Skipped 3 previous similar messages [ 269.567115] Lustre: server umount lustre-MDT0000 complete [ 270.422963] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 270.431609] LustreError: Skipped 2 previous similar messages [ 270.714118] LustreError: 20737:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761264 with bad export cookie 3086013104066360461 [ 270.715577] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 270.723572] LustreError: 20737:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 270.829559] Lustre: server umount lustre-MDT0001 complete [ 282.035445] Lustre: server umount lustre-OST0000 complete [ 286.919628] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 289.437713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 289.572023] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 289.590693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 289.593496] Lustre: Skipped 2 previous similar messages [ 290.484724] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 292.913069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 293.864918] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 296.238397] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 296.243388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 297.723751] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 300.283420] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 300.287542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 300.314846] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 300.320239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 300.367059] Lustre: lustre-OST0001: new disk, initializing [ 300.369503] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 301.891676] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 301.920092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 301.923612] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 301.942754] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 306.261541] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 307.409130] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 317.959442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.961095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.966657] Lustre: Skipped 5 previous similar messages [ 322.002555] Lustre: server umount lustre-MDT0000 complete [ 322.967139] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.971789] LustreError: Skipped 3 previous similar messages [ 323.305130] LustreError: 26376:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761316 with bad export cookie 3086013104066361308 [ 323.307053] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 323.319455] LustreError: 26376:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 323.491539] Lustre: server umount lustre-MDT0001 complete [ 334.917011] Lustre: server umount lustre-OST0000 complete [ 346.392514] Lustre: server umount lustre-OST0001 complete [ 351.550032] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 354.514870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 354.686299] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 354.719725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 354.726013] Lustre: Skipped 3 previous similar messages [ 355.771619] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 358.553053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 359.714612] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 362.441891] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 362.446532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 364.035199] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 366.532740] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 366.537510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 368.056693] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 374.456596] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 375.406051] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 375.408549] Lustre: Skipped 1 previous similar message [ 383.718105] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 383.720948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.725547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 383.728372] Lustre: Skipped 3 previous similar messages [ 387.643797] Lustre: server umount lustre-MDT0000 complete [ 388.683460] LustreError: 31787:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761382 with bad export cookie 3086013104066362554 [ 388.684728] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 388.691676] LustreError: 31787:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 388.807975] Lustre: server umount lustre-MDT0001 complete [ 399.918151] Lustre: server umount lustre-OST0000 complete [ 411.005172] Lustre: server umount lustre-OST0001 complete [ 414.857164] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 11:03:27 (1712761407) [ 421.396368] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 423.833401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.955044] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 423.961794] LustreError: Skipped 1 previous similar message [ 424.788703] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 427.160935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 428.115431] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 430.540429] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 430.544859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 432.105576] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 434.463499] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 434.468709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 435.933859] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 442.243432] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 443.178478] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 452.294297] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 452.297105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.301681] Lustre: Skipped 3 previous similar messages [ 452.303860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 452.306062] Lustre: Skipped 4 previous similar messages [ 455.190713] Lustre: server umount lustre-MDT0000 complete [ 456.318328] LustreError: 9598:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761449 with bad export cookie 3086013104066363625 [ 456.319472] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 456.327770] LustreError: 9598:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 456.434945] Lustre: server umount lustre-MDT0001 complete [ 467.672520] Lustre: server umount lustre-OST0000 complete [ 478.894873] Lustre: server umount lustre-OST0001 complete [ 483.325252] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 11:04:35 (1712761475) [ 486.884003] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 489.289304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.401208] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.407238] LustreError: Skipped 1 previous similar message [ 489.415694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 489.418493] Lustre: Skipped 7 previous similar messages [ 490.173908] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 492.424064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 493.310729] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 495.554664] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 495.559033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 496.946346] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 499.267441] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 499.274521] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 500.673960] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 507.070330] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 517.366274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.371173] Lustre: Skipped 5 previous similar messages [ 520.185753] Lustre: server umount lustre-MDT0000 complete [ 521.205324] LustreError: 16195:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761514 with bad export cookie 3086013104066364696 [ 521.206657] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.213010] LustreError: 16195:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 527.322408] Lustre: server umount lustre-MDT0001 complete [ 529.590725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.382688] LustreError: 3116:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88009b6ab800 x1795960126716992/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 532.285430] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 534.578398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 535.514475] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 536.630018] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 550.695918] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 555.700924] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 555.703686] Lustre: Skipped 1 previous similar message [ 557.380497] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 20 sec [ 558.015102] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 558.424281] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 559.701994] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 565.481896] Lustre: server umount lustre-MDT0000 complete [ 566.554505] LustreError: 16195:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761559 with bad export cookie 3086013104066365809 [ 566.556136] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 566.562004] LustreError: 16195:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 566.658935] Lustre: server umount lustre-MDT0001 complete [ 577.760721] Lustre: server umount lustre-OST0000 complete [ 588.844779] Lustre: server umount lustre-OST0001 complete [ 592.590737] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 11:06:25 (1712761585) [ 595.895086] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 598.492049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 599.552666] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 601.901389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 602.886042] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 605.237281] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 605.240760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 606.716775] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 609.205013] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 609.210437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 610.560541] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 616.887294] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 617.855834] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 617.858760] Lustre: Skipped 2 previous similar messages [ 626.998883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 627.001523] Lustre: Skipped 22 previous similar messages [ 632.596640] Lustre: server umount lustre-MDT0000 complete [ 633.640048] LustreError: 26548:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761626 with bad export cookie 3086013104066366614 [ 633.641015] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.648202] LustreError: 26548:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 633.749191] Lustre: server umount lustre-MDT0001 complete [ 644.760485] Lustre: server umount lustre-OST0000 complete [ 655.790873] Lustre: server umount lustre-OST0001 complete [ 659.314423] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 11:07:31 (1712761651) [ 659.915943] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 661.862899] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 11:07:34 (1712761654) [ 664.117477] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 665.516576] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 667.757366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 669.354128] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 669.360109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 671.237436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 671.262000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 671.340918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 671.375858] Lustre: lustre-MDT0000: new disk, initializing [ 671.405107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 672.270621] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 674.600899] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 676.462700] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 676.466513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 676.482847] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 676.487190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 677.860156] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 678.540097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 678.547757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 680.158289] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 680.647465] Lustre: DEBUG MARKER: create test files [ 688.549924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 688.554094] Lustre: Skipped 11 previous similar messages [ 688.555629] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 693.557777] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 693.561775] LustreError: Skipped 13 previous similar messages [ 693.643669] Lustre: server umount lustre-OST0000 complete [ 694.688619] Lustre: server umount lustre-MDT0000 complete [ 697.546566] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 698.029336] Key type lgssc unregistered [ 698.095920] LNet: 1195:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.098415] LNet: Removed LNI 192.168.202.133@tcp [ 705.876340] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 706.025856] alg: No test for adler32 (adler32-zlib) [ 706.779460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 706.846480] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 706.898555] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 706.900660] LNet: Accept secure, port 988 [ 708.430428] Key type lgssc registered [ 708.644946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 710.583291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 710.590400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.723130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 712.533116] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 713.656664] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 715.547445] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 715.552070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 715.659487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 716.983633] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 718.135908] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 722.660710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 722.684704] Lustre: 4158:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 722.688526] Lustre: 4158:0:(osd_handler.c:1975:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 722.691875] Lustre: 4158:0:(osd_handler.c:1982:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 722.695729] Lustre: 4158:0:(osd_handler.c:1992:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 722.699517] Lustre: 4158:0:(osd_handler.c:1999:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 722.702470] Lustre: 4158:0:(osd_handler.c:2006:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.938844] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 727.380196] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 732.677964] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 732.678010] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 732.678313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 734.859499] Lustre: server umount lustre-OST0000 complete [ 735.983130] Lustre: server umount lustre-MDT0000 complete [ 738.935881] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 739.454252] Key type lgssc unregistered [ 739.517739] LNet: 5803:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.520845] LNet: Removed LNI 192.168.202.133@tcp [ 748.144471] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 748.320215] alg: No test for adler32 (adler32-zlib) [ 749.072424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 749.142671] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 749.197976] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 749.200817] LNet: Accept secure, port 988 [ 750.736459] Key type lgssc registered [ 750.966637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 752.901881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 752.909105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 754.036760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 754.851363] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 755.962524] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 757.818961] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 757.824229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 757.923394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 759.285886] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 760.414105] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 763.923222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 763.943755] Lustre: 8896:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 763.946966] Lustre: 8896:0:(osd_handler.c:1975:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 763.949594] Lustre: 8896:0:(osd_handler.c:1982:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 763.952278] Lustre: 8896:0:(osd_handler.c:1992:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 763.955345] Lustre: 8896:0:(osd_handler.c:1999:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 763.957905] Lustre: 8896:0:(osd_handler.c:2006:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 768.203348] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 771.989518] Lustre: server umount lustre-OST0000 complete [ 783.110544] Lustre: server umount lustre-MDT0000 complete [ 786.052397] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 786.540469] Key type lgssc unregistered [ 786.604675] LNet: 10484:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.607386] LNet: Removed LNI 192.168.202.133@tcp [ 791.125659] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 795.595023] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 795.749412] alg: No test for adler32 (adler32-zlib) [ 796.500394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 796.572278] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 796.628099] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 796.630250] LNet: Accept secure, port 988 [ 798.164443] Key type lgssc registered [ 798.432003] Lustre: Echo OBD driver; http://www.lustre.org/ [ 800.819604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 802.655725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 804.394836] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 804.400862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 806.201042] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 806.205830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 808.164050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 808.182754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 808.189223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.280951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 809.290561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 809.323330] Lustre: lustre-MDT0000: new disk, initializing [ 809.344478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 809.350808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 810.189360] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 813.743939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 813.760365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 813.787595] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 813.803520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 813.805733] Lustre: Skipped 1 previous similar message [ 813.841753] Lustre: lustre-MDT0001: new disk, initializing [ 813.867073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 813.876040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 813.879183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 814.669876] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 817.042296] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 817.729454] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 819.608880] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 819.612496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 819.627759] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 819.631786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 819.727044] Lustre: lustre-OST0000: new disk, initializing [ 819.729512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 819.751169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 821.045233] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 823.331902] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 824.111333] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 824.287854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 824.291224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 824.307713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 825.173094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 825.932621] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 825.978065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 829.318249] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 829.318533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 829.318824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 829.326239] LustreError: Skipped 1 previous similar message [ 832.534354] Lustre: server umount lustre-OST0000 complete [ 833.893962] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 833.896370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 833.899989] Lustre: Skipped 1 previous similar message [ 833.901679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 833.903628] Lustre: Skipped 1 previous similar message [ 839.333873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 839.336298] Lustre: Skipped 3 previous similar messages [ 839.528788] Lustre: server umount lustre-MDT0000 complete [ 840.538970] LustreError: 17778:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761833 with bad export cookie 7574438250468021492 [ 840.540154] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 840.548698] LustreError: 17778:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 840.677120] Lustre: server umount lustre-MDT0001 complete [ 844.430406] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 11:10:36 (1712761836) [ 846.365308] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 849.467776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 855.272969] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 856.825147] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 858.900038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 860.512414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 861.982819] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 861.986750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 863.471070] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 863.475379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 865.226054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 865.242516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 865.311035] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 865.320044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 865.353535] Lustre: lustre-MDT0000: new disk, initializing [ 865.371937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 865.378139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 866.150113] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 869.533178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 869.554174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 869.578417] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 869.590398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 869.592640] Lustre: Skipped 1 previous similar message [ 869.624548] Lustre: lustre-MDT0001: new disk, initializing [ 869.644999] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 869.653594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 869.656881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 870.393530] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 872.596146] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 873.123752] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 874.769677] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 874.772481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 874.791149] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 874.795580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 876.054401] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 878.276728] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 879.008198] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 879.821455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 879.824943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 879.841631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 880.066586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 880.797462] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 880.845988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 884.838019] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 884.838349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.838350] Lustre: Skipped 1 previous similar message [ 884.838809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 884.848942] LustreError: Skipped 1 previous similar message [ 887.384196] Lustre: server umount lustre-OST0000 complete [ 889.669857] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 889.672979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.677608] Lustre: Skipped 1 previous similar message [ 889.679592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 889.681392] Lustre: Skipped 1 previous similar message [ 894.386258] Lustre: server umount lustre-MDT0000 complete [ 894.853808] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 894.859145] LustreError: Skipped 1 previous similar message [ 895.328145] LustreError: 23615:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761888 with bad export cookie 7574438250468022332 [ 895.329081] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 895.335059] LustreError: 23615:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 895.446027] Lustre: server umount lustre-MDT0001 complete [ 898.895687] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 11:11:31 (1712761891) [ 900.563988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 900.689119] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 900.703543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.706188] Lustre: Skipped 1 previous similar message [ 901.441948] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 903.571296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 904.430889] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 905.462399] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 905.991241] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 907.647963] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 907.651147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 908.901542] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 909.886079] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 910.781571] Lustre: 31417:0:(genops.c:1659:obd_export_evict_by_uuid()) MGS: evicting 5f5dd423-2a16-40cd-be74-3910bc3781aa at adminstrative request [ 912.734376] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 912.739637] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 922.740015] Lustre: MGS: Client 5f5dd423-2a16-40cd-be74-3910bc3781aa (at 0@lo) reconnecting [ 922.742689] Lustre: Evicted from MGS (at 192.168.202.133@tcp) after server handle changed from 0x691dcf752c471b84 to 0x691dcf752c471dfa [ 922.747196] Lustre: MGC192.168.202.133@tcp: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 942.559965] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 942.564654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 942.586942] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 942.590638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 942.627444] Lustre: lustre-OST0001: new disk, initializing [ 942.629373] Lustre: Skipped 1 previous similar message [ 942.631815] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 942.634452] Lustre: Skipped 1 previous similar message [ 942.653431] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 942.655993] Lustre: Skipped 2 previous similar messages [ 943.929324] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 946.186476] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 946.938791] Lustre: server umount lustre-OST0001 complete [ 950.805821] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 950.808592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 950.812460] Lustre: Skipped 1 previous similar message [ 950.814441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 950.816373] Lustre: Skipped 1 previous similar message [ 951.529251] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 953.975357] Lustre: server umount lustre-OST0000 complete [ 957.008539] Lustre: server umount lustre-MDT0000 complete [ 957.951049] LustreError: 28725:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761951 with bad export cookie 7574438250468023802 [ 957.952308] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 957.958597] LustreError: 28725:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 958.047645] Lustre: server umount lustre-MDT0001 complete [ 961.450613] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 11:12:33 (1712761953) [ 961.823400] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 963.744084] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 11:12:36 (1712761956) [ 964.117077] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 966.041593] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 11:12:38 (1712761958) [ 966.417312] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 968.344606] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 11:12:40 (1712761960) [ 968.720809] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 969.885614] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 969.888979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 970.152473] loop: module loaded [ 970.155889] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 970.161931] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 970.535382] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 971.033152] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 971.909048] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 974.030473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 974.959323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 975.941924] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 975.945517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 976.953783] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 976.957241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 978.739633] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 978.743686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 978.760974] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 978.765189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 978.771824] LustreError: 6030:0:(obd_config.c:797:class_setup()) setup lustre-OST0000-osd failed (-28) [ 978.774575] LustreError: 6030:0:(obd_mount.c:215:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 978.777389] LustreError: 6030:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 978.781365] LustreError: 6030:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 983.293721] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 984.919285] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 986.987410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 988.561005] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 990.019424] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 990.022649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 991.537480] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 991.541206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 993.208788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 993.222777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 993.282727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 993.290627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 993.322078] Lustre: lustre-MDT0000: new disk, initializing [ 993.337902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.343276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 994.062540] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 997.415274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 997.430334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 997.521169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 998.259456] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1000.465752] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1000.991441] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1002.656554] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1002.659957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1002.677276] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1002.680738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1003.926591] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1006.127698] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1006.905889] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1012.023420] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1012.038544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1013.034452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 1013.829355] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1013.875029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1017.029948] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1017.032799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.036564] Lustre: Skipped 3 previous similar messages [ 1017.038391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1017.040282] Lustre: Skipped 4 previous similar messages [ 1020.420287] Lustre: server umount lustre-OST0000 complete [ 1027.481610] Lustre: server umount lustre-MDT0000 complete [ 1028.543079] LustreError: 9532:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762021 with bad export cookie 7574438250468024341 [ 1028.544449] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1028.552924] LustreError: 9532:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1028.652509] Lustre: server umount lustre-MDT0001 complete [ 1032.327961] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 11:13:44 (1712762024) [ 1034.049531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1034.180808] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1034.186431] LustreError: Skipped 1 previous similar message [ 1034.983621] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1037.240145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1038.139672] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1039.210825] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1039.817500] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1041.580178] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1041.584545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1042.969781] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1044.065288] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1044.922181] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1044.979695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1045.780250] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1047.856105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 1049.594026] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1049.597100] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1049.616871] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1049.619796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1049.656621] Lustre: lustre-OST0001: new disk, initializing [ 1049.658057] Lustre: Skipped 2 previous similar messages [ 1049.659797] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1049.662136] Lustre: Skipped 3 previous similar messages [ 1050.974869] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1053.003506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1053.006830] Lustre: Skipped 2 previous similar messages [ 1053.008371] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1053.020585] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1053.326622] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1054.111986] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1054.164486] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1054.973783] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1055.028324] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1063.878474] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1068.783554] Lustre: Failing over lustre-MDT0000 [ 1068.891209] Lustre: server umount lustre-MDT0000 complete [ 1071.245042] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1071.250238] LustreError: Skipped 1 previous similar message [ 1072.373912] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1072.376811] LustreError: Skipped 1 previous similar message [ 1072.378158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.382063] Lustre: Skipped 3 previous similar messages [ 1081.255522] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.260815] LustreError: Skipped 12 previous similar messages [ 1082.375104] LDISKFS-fs (dm-0): recovery complete [ 1082.376580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.406834] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1082.480018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 1082.482893] Lustre: Skipped 6 previous similar messages [ 1082.495551] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1083.252330] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1086.263092] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1086.263289] Lustre: 19452:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1087.493835] Lustre: 14719:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1087.498106] Lustre: 14719:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 1 previous similar message [ 1087.503230] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1087.514550] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1087.817382] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1088.122924] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1088.125983] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 1 previous similar message [ 1088.428427] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1088.431421] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 1 previous similar message [ 1088.836409] Lustre: 21305:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1088.841689] Lustre: 21305:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 8 previous similar messages [ 1089.245043] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1089.249886] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 2 previous similar messages [ 1089.553459] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1089.557323] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 2 previous similar messages [ 1091.167692] Lustre: 21305:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1091.172619] Lustre: 21305:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 5 previous similar messages [ 1091.574364] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1091.577121] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 4 previous similar messages [ 1091.580093] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1091.583232] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 5 previous similar messages [ 1095.490896] Lustre: 21305:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1095.494210] Lustre: 21305:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 10 previous similar messages [ 1095.896368] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1095.899195] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 10 previous similar messages [ 1095.903714] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1095.906887] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 10 previous similar messages [ 1103.821174] Lustre: 21305:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1103.826153] Lustre: 21305:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 20 previous similar messages [ 1104.129371] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1104.132766] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 20 previous similar messages [ 1104.135598] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1104.138915] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 20 previous similar messages [ 1119.967147] Lustre: 21305:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1119.971513] Lustre: 21305:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 40 previous similar messages [ 1120.373360] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1120.375861] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 40 previous similar messages [ 1120.379755] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1120.383059] LustreError: 21305:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 40 previous similar messages [ 1146.123429] Lustre: 21305:0:(ldlm_lib.c:2300:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1146.126605] Lustre: 21305:0:(ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1146.128930] Lustre: 21305:0:(genops.c:1516:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 8f4e5534-b25b-4058-a1ec-992eb64c541b@192.168.202.33@tcp [ 1146.132566] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1146.134906] Lustre: 21305:0:(ldlm_lib.c:1824:abort_req_replay_queue()) @@@ aborted: req@ffff8800a9802d80 x1795960900448384/t0(8589934746) o35->8f4e5534-b25b-4058-a1ec-992eb64c541b@192.168.202.33@tcp:105/0 lens 392/0 e 0 to 0 dl 1712762150 ref 1 fl Complete:/204/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 1146.143144] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1146.146715] Lustre: lustre-MDT0000: Denying connection for new client 8f4e5534-b25b-4058-a1ec-992eb64c541b (at 192.168.202.33@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 1146.149630] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1146.155918] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1146.156025] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1146.162286] Lustre: Skipped 2 previous similar messages [ 1146.169706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:72 to 0x280000401:577) [ 1146.169720] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:39 to 0x2c0000401:545) [ 1151.317995] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1151.322309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.328850] Lustre: Skipped 3 previous similar messages [ 1151.331698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1151.335143] Lustre: Skipped 5 previous similar messages [ 1155.546378] Lustre: server umount lustre-OST0000 complete [ 1157.606030] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.609875] LustreError: Skipped 1 previous similar message [ 1162.662803] Lustre: server umount lustre-OST0001 complete [ 1169.849158] Lustre: server umount lustre-MDT0000 complete [ 1170.800099] LustreError: 16571:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762164 with bad export cookie 7574438250468075483 [ 1170.801322] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1170.807882] LustreError: 16571:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1170.910889] Lustre: server umount lustre-MDT0001 complete [ 1174.270608] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 11:16:06 (1712762166) [ 1176.072099] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1176.075156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1176.077809] Lustre: *** cfs_fail_loc=197, val=0*** [ 1193.129821] LustreError: 24242:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 1: rc = -5 [ 1209.132732] LustreError: 24242:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 1335.142949] Lustre: *** cfs_fail_loc=197, val=0*** [ 1335.144227] Lustre: Skipped 65 previous similar messages [ 1358.153542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.444121] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1360.498138] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1361.057321] Lustre: server umount lustre-OST0000 complete [ 1364.614541] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 11:19:17 (1712762357) [ 1377.194341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1377.345805] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1378.134210] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1380.403141] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1382.134999] LustreError: 26621:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1382.138775] LustreError: 26621:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1382.141143] LustreError: 26621:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 1382.243494] Lustre: server umount lustre-MDT0000 complete [ 1386.085565] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1386.088921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1390.080297] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 1391.682053] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1393.827332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1395.460487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1396.941692] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1396.944869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1398.466215] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1398.469455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1400.276540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1400.296905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1400.374641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1400.377962] Lustre: Skipped 1 previous similar message [ 1400.385962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1400.420580] Lustre: lustre-MDT0000: new disk, initializing [ 1400.442402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1401.192384] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1404.587412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1404.606645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1404.704485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1405.505549] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1407.775785] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1408.340791] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1410.058643] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1410.061956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1410.086333] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1410.090294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1411.434748] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1411.541351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1413.724666] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1414.560899] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1414.613570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1415.410867] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1415.465797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1416.533999] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1416.536314] LustreError: Skipped 1 previous similar message [ 1416.538111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.541513] Lustre: Skipped 5 previous similar messages [ 1416.543054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1416.545153] Lustre: Skipped 7 previous similar messages [ 1422.030892] Lustre: server umount lustre-OST0000 complete [ 1429.070398] Lustre: server umount lustre-MDT0000 complete [ 1430.039054] LustreError: 31566:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762423 with bad export cookie 7574438250468085787 [ 1430.040428] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1430.045539] LustreError: 31566:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1430.146797] Lustre: server umount lustre-MDT0001 complete [ 1433.761504] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 11:20:26 (1712762426) [ 1436.073386] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1436.535590] Key type lgssc unregistered [ 1436.600701] LNet: 4187:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.604597] LNet: Removed LNI 192.168.202.133@tcp [ 1440.976380] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 1445.250557] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1445.377551] alg: No test for adler32 (adler32-zlib) [ 1446.128406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1446.189422] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1446.241097] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1446.242964] LNet: Accept secure, port 988 [ 1447.774577] Key type lgssc registered [ 1448.030960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1450.197445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1451.826320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1453.355852] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1453.358958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1454.880843] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1454.884068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1456.260750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1457.961552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1457.978888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1457.982599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.071181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1459.079697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1459.118020] Lustre: lustre-MDT0000: new disk, initializing [ 1459.137463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1459.144045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1459.905539] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1463.306871] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1463.309979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1463.328128] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1463.331484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1463.405056] Lustre: lustre-OST0000: new disk, initializing [ 1463.407204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1463.408941] Lustre: Skipped 1 previous similar message [ 1463.421989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1464.516228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1464.519157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1464.527888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1464.649920] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1468.068663] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1468.071887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1468.086790] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1468.089952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1468.120895] Lustre: lustre-OST0001: new disk, initializing [ 1468.122935] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1468.140870] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1469.385804] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1473.141996] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1473.147355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1473.149605] Lustre: Skipped 1 previous similar message [ 1478.149923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1478.151934] Lustre: Skipped 1 previous similar message [ 1479.008310] Lustre: server umount lustre-MDT0000 complete [ 1480.682668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1481.881921] LustreError: 10786:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762475 with bad export cookie 1918833778337113958 [ 1481.883028] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1481.888141] LustreError: 10786:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1481.895731] Lustre: server umount lustre-OST0000 complete [ 1482.933423] Lustre: server umount lustre-OST0001 complete [ 1486.524419] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 11:21:18 (1712762478) [ 1488.162490] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1490.106019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1490.391984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1491.755394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1492.044942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1495.153921] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 11:21:27 (1712762487) [ 1498.242512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1498.769765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1499.285275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1499.777807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1500.040607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1500.554113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1504.647680] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 1506.146550] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1508.370443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1510.027927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1511.592456] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1511.597216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1513.186139] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1513.190383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1514.958250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1514.980680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.053170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1515.061075] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1515.098090] Lustre: lustre-MDT0000: new disk, initializing [ 1515.118591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1515.124407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1515.872188] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1519.290998] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1519.313618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1519.338737] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1519.352006] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1519.353747] Lustre: Skipped 1 previous similar message [ 1519.388531] Lustre: lustre-MDT0001: new disk, initializing [ 1519.409447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1519.418667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1519.421686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1520.193082] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1522.511499] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1523.107750] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1524.865567] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1524.868655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1524.886021] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1524.889090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1526.224236] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1528.530953] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1529.342124] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1529.539554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1529.542205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1529.557968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1530.407203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1531.157970] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1531.207836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1534.566657] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1534.566790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.566792] Lustre: Skipped 1 previous similar message [ 1534.567443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1534.579221] LustreError: Skipped 1 previous similar message [ 1537.787815] Lustre: server umount lustre-OST0000 complete [ 1539.430160] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1539.433692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.438522] Lustre: Skipped 1 previous similar message [ 1539.440800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1539.442815] Lustre: Skipped 1 previous similar message [ 1544.581983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1544.584487] Lustre: Skipped 2 previous similar messages [ 1544.824565] Lustre: server umount lustre-MDT0000 complete [ 1545.824166] LustreError: 19332:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762539 with bad export cookie 1918833778337115064 [ 1545.825348] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1545.831292] LustreError: 19332:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1545.948566] Lustre: server umount lustre-MDT0001 complete [ 1549.492475] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 11:22:21 (1712762541) [ 1551.186407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1551.319365] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1551.339464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.341674] Lustre: Skipped 1 previous similar message [ 1552.106504] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1554.359902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1555.195041] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1556.220872] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1556.776046] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1558.517037] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1558.520060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1559.794967] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1560.849518] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1567.650448] Lustre: *** cfs_fail_loc=159, val=0*** [ 1583.656768] Lustre: lustre-MDT0000: Client 21d31af7-50b2-492f-b8b7-662f0ad822eb (at 192.168.202.33@tcp) reconnecting [ 1583.925079] Lustre: *** cfs_fail_loc=159, val=0*** [ 1583.926897] Lustre: Skipped 6 previous similar messages [ 1599.925366] Lustre: lustre-MDT0000: Client 21d31af7-50b2-492f-b8b7-662f0ad822eb (at 192.168.202.33@tcp) reconnecting [ 1601.670081] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1601.673903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1601.678551] Lustre: Skipped 1 previous similar message [ 1601.681162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1604.678139] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1606.691898] Lustre: server umount lustre-OST0000 complete [ 1609.526179] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1609.530135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.535776] Lustre: Skipped 1 previous similar message [ 1613.724973] Lustre: server umount lustre-MDT0000 complete [ 1614.693792] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.697410] LustreError: Skipped 1 previous similar message [ 1614.703089] LustreError: 26319:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762608 with bad export cookie 1918833778337115904 [ 1614.704144] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1614.709477] LustreError: 26319:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1614.810774] Lustre: server umount lustre-MDT0001 complete [ 1617.657567] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1618.128218] Key type lgssc unregistered [ 1618.194630] LNet: 28373:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1618.197156] LNet: Removed LNI 192.168.202.133@tcp [ 1626.072023] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1626.205535] alg: No test for adler32 (adler32-zlib) [ 1626.956393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1627.015202] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1627.068223] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1627.069825] LNet: Accept secure, port 988 [ 1628.595454] Key type lgssc registered [ 1628.838565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1630.785722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1630.792637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1631.947863] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1631.964606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1632.774390] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1634.977845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1635.077660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1635.897246] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1637.087472] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1637.765162] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1639.472533] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1639.476403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1639.576090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1640.788673] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1641.805723] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1642.578277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 1652.100480] Lustre: *** cfs_fail_loc=159, val=0*** [ 1668.114408] Lustre: lustre-MDT0000: Client f1276144-6bf5-4039-8386-26f6e6f2e973 (at 192.168.202.33@tcp) reconnecting [ 1668.504784] Lustre: *** cfs_fail_loc=159, val=0*** [ 1684.515118] Lustre: lustre-MDT0001: Client f1276144-6bf5-4039-8386-26f6e6f2e973 (at 192.168.202.33@tcp) reconnecting [ 1684.764833] Lustre: *** cfs_fail_loc=159, val=0*** [ 1684.766228] Lustre: Skipped 4 previous similar messages [ 1700.765502] Lustre: lustre-MDT0001: Client f1276144-6bf5-4039-8386-26f6e6f2e973 (at 192.168.202.33@tcp) reconnecting [ 1701.483504] Lustre: 961:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1702.069655] Lustre: *** cfs_fail_loc=159, val=0*** [ 1702.071305] Lustre: Skipped 3 previous similar messages [ 1718.072615] Lustre: lustre-MDT0000: Client 1da0df40-3f11-43e1-9b14-4f8e284df8a7 (at 192.168.202.33@tcp) reconnecting [ 1718.331901] Lustre: *** cfs_fail_loc=159, val=0*** [ 1718.333312] Lustre: Skipped 14 previous similar messages [ 1734.335511] Lustre: lustre-MDT0000: Client 1da0df40-3f11-43e1-9b14-4f8e284df8a7 (at 192.168.202.33@tcp) reconnecting [ 1734.591416] Lustre: 1313:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1735.718069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1735.718230] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1735.718632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1735.727567] Lustre: Skipped 1 previous similar message [ 1740.725858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1740.727630] Lustre: Skipped 2 previous similar messages [ 1741.529863] Lustre: server umount lustre-OST0000 complete [ 1745.238004] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1745.241721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.247571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1748.594850] Lustre: server umount lustre-MDT0000 complete [ 1749.562237] LustreError: 30249:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762742 with bad export cookie 9402934559158672505 [ 1749.563696] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1749.570275] LustreError: 30249:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1749.675712] Lustre: server umount lustre-MDT0001 complete [ 1752.492518] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1752.952901] Key type lgssc unregistered [ 1753.017579] LNet: 2511:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.019623] LNet: Removed LNI 192.168.202.133@tcp [ 1760.593607] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1760.698718] alg: No test for adler32 (adler32-zlib) [ 1761.449348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1761.504670] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1761.559985] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1761.562510] LNet: Accept secure, port 988 [ 1763.098466] Key type lgssc registered [ 1763.309024] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1765.164440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1765.169489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.302384] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1766.313295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1767.028785] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1769.114402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1769.207765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1769.943817] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1770.932004] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1771.470924] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1773.115463] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1773.118081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1773.206280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.366329] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1775.371463] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1776.207624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 1779.207686] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 1783.172965] Lustre: 6972:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1783.915980] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1784.682250] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1784.686145] Lustre: Skipped 1 previous similar message [ 1785.145632] Lustre: 7364:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1785.150042] Lustre: 7364:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) Skipped 1 previous similar message [ 1786.229924] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1786.231830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.235532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1789.222028] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1789.222341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1789.228603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1791.855534] Lustre: server umount lustre-OST0000 complete [ 1794.230089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.230119] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1794.232912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1794.232914] Lustre: Skipped 1 previous similar message [ 1794.245304] Lustre: Skipped 1 previous similar message [ 1798.921007] Lustre: server umount lustre-MDT0000 complete [ 1799.237803] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1799.244249] LustreError: Skipped 1 previous similar message [ 1799.929275] LustreError: 4398:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762793 with bad export cookie 564073575145550315 [ 1799.930332] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1799.939290] LustreError: 4398:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1800.050817] Lustre: server umount lustre-MDT0001 complete [ 1802.963899] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1803.481463] Key type lgssc unregistered [ 1803.551781] LNet: 8569:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.554442] LNet: Removed LNI 192.168.202.133@tcp [ 1811.578295] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1811.726676] alg: No test for adler32 (adler32-zlib) [ 1812.478351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1812.543761] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1812.597681] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1812.600282] LNet: Accept secure, port 988 [ 1814.133423] Key type lgssc registered [ 1814.341172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1816.302067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1816.308092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1817.441336] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1817.455175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1818.246522] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1820.514847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1820.617458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1821.419744] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1822.532202] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1823.138474] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1824.943181] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1824.947616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1825.050285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1826.347565] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1827.049955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 1827.466915] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1830.051351] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 1834.044133] Lustre: *** cfs_fail_loc=159, val=0*** [ 1850.048681] Lustre: lustre-MDT0000: Client 3674c2ab-99ee-4c42-bb0b-848747ea7714 (at 192.168.202.33@tcp) reconnecting [ 1852.085918] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1852.088654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1852.092046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1855.094151] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1855.094366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1855.098336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.869698] Lustre: server umount lustre-OST0000 complete [ 1860.102025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1860.102339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1860.102340] Lustre: Skipped 1 previous similar message [ 1860.108414] Lustre: Skipped 1 previous similar message [ 1863.944569] Lustre: server umount lustre-MDT0000 complete [ 1864.948470] LustreError: 10447:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762858 with bad export cookie 17049847862075784035 [ 1864.949832] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1864.954994] LustreError: 10447:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1865.066739] Lustre: server umount lustre-MDT0001 complete [ 1867.950494] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1868.438212] Key type lgssc unregistered [ 1868.506771] LNet: 14239:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1868.509482] LNet: Removed LNI 192.168.202.133@tcp [ 1876.512914] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1876.667788] alg: No test for adler32 (adler32-zlib) [ 1877.419379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1877.487251] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1877.544367] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1877.546628] LNet: Accept secure, port 988 [ 1879.077434] Key type lgssc registered [ 1879.296194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1881.285069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1881.290686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1882.418564] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1882.433358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.223716] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1885.476956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1885.573900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1886.359880] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1887.441668] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1888.034817] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1889.804375] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1889.807564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1889.903834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.104629] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1891.905597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 1892.189882] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1894.905234] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 1909.913516] Lustre: lustre-OST0000: evicting ce588138-c3e3-4d1b-93d1-247432af3d00 (at 192.168.202.33@tcp) by administrative request [ 1909.916839] Lustre: Skipped 1 previous similar message [ 1934.965979] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1934.966115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.966491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1939.441699] Lustre: server umount lustre-OST0000 complete [ 1940.645957] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1940.649052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.653888] Lustre: Skipped 1 previous similar message [ 1940.655596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1940.657058] Lustre: Skipped 1 previous similar message [ 1944.982019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1944.982075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1944.990100] Lustre: Skipped 1 previous similar message [ 1946.503660] Lustre: server umount lustre-MDT0000 complete [ 1947.497168] LustreError: 16114:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762940 with bad export cookie 16435588936875345716 [ 1947.498149] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1947.502790] LustreError: 16114:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1947.611663] Lustre: server umount lustre-MDT0001 complete [ 1950.499674] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1951.029251] Key type lgssc unregistered [ 1951.098624] LNet: 20432:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.101200] LNet: Removed LNI 192.168.202.133@tcp [ 1979.163427] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1979.338460] alg: No test for adler32 (adler32-zlib) [ 1980.089428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1980.154021] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1980.203925] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1980.206612] LNet: Accept secure, port 988 [ 1981.738464] Key type lgssc registered [ 1981.962869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1983.993669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1983.998329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1985.136644] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1985.150704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1985.967508] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1988.238419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1988.341150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1989.162438] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1990.217392] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1990.809324] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1992.607528] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1992.611888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1992.715359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.983171] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1994.715308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1995.036036] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1997.716228] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 2002.725973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.726321] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2002.731021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2002.734556] Lustre: Skipped 1 previous similar message [ 2003.992302] Lustre: server umount lustre-OST0000 complete [ 2007.062281] Lustre: server umount lustre-MDT0000 complete [ 2007.733848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.734188] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.743633] Lustre: Skipped 2 previous similar messages [ 2008.102295] LustreError: 26072:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763001 with bad export cookie 8341982326084888428 [ 2008.103356] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2008.109775] LustreError: 26072:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2008.221284] Lustre: server umount lustre-MDT0001 complete [ 2011.225253] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2011.763351] Key type lgssc unregistered [ 2011.833691] LNet: 29715:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.837742] LNet: Removed LNI 192.168.202.133@tcp [ 2019.826359] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2019.979181] alg: No test for adler32 (adler32-zlib) [ 2020.730375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2020.789898] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 2020.840726] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2020.842110] LNet: Accept secure, port 988 [ 2022.373462] Key type lgssc registered [ 2022.603885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2024.345226] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2024.348961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2027.809776] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 11:30:20 (1712763020) [ 2031.748127] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 2033.434574] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2035.843347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2037.587888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2039.179741] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2039.183701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2040.824149] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2040.828844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2042.686198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2042.702827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2042.708372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2043.786594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2043.795799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2043.832644] Lustre: lustre-MDT0000: new disk, initializing [ 2043.850450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.858296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2044.660642] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2048.219391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2048.239316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2048.268641] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2048.283668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2048.285393] Lustre: Skipped 1 previous similar message [ 2048.321090] Lustre: lustre-MDT0001: new disk, initializing [ 2048.345432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2048.356413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2048.359295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2049.142379] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2051.465106] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2052.053839] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2053.842586] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2053.846228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2053.864645] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2053.867801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2053.957667] Lustre: lustre-OST0000: new disk, initializing [ 2053.959691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2053.981031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.276728] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2057.614492] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2060.279376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2060.282578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2060.303018] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2065.301937] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2065.302148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2065.302774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2065.310653] LustreError: Skipped 1 previous similar message [ 2067.345242] Lustre: server umount lustre-OST0000 complete [ 2068.373768] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2068.375952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2068.379742] Lustre: Skipped 1 previous similar message [ 2068.381818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2068.383678] Lustre: Skipped 1 previous similar message [ 2068.435446] Lustre: server umount lustre-MDT0000 complete [ 2069.439168] LustreError: 3059:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763062 with bad export cookie 10290507515124867704 [ 2069.441285] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2069.448491] LustreError: 3059:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2075.566072] Lustre: server umount lustre-MDT0001 complete [ 2078.474288] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2078.998657] Key type lgssc unregistered [ 2079.070643] LNet: 7137:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.072750] LNet: Removed LNI 192.168.202.133@tcp [ 2086.973380] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2087.132322] alg: No test for adler32 (adler32-zlib) [ 2087.884357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2087.947248] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 2087.993457] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2087.994716] LNet: Accept secure, port 988 [ 2089.524423] Key type lgssc registered [ 2089.738964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2091.642373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2091.648914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2092.788049] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2092.800196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.587239] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2095.825059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2095.935809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2096.718552] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2097.814631] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2098.432980] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2100.208639] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2100.211877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2100.312117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.592583] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2102.663886] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2103.314737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2110.627415] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2110.634227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.639175] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2110.641791] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2110.646656] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2111.140551] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2111.142919] Lustre: Skipped 1 previous similar message [ 2111.147469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2111.150584] Lustre: Skipped 1 previous similar message [ 2111.151773] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2111.154323] Lustre: Skipped 1 previous similar message [ 2111.155490] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2111.158226] LustreError: Skipped 1 previous similar message [ 2111.159543] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2111.161278] Lustre: Skipped 1 previous similar message [ 2112.360050] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2112.362840] Lustre: Skipped 4 previous similar messages [ 2112.369413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.374538] Lustre: Skipped 4 previous similar messages [ 2112.376314] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2112.378073] Lustre: Skipped 4 previous similar messages [ 2112.379214] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2112.381996] LustreError: Skipped 4 previous similar messages [ 2112.383299] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2112.384946] Lustre: Skipped 4 previous similar messages [ 2114.563584] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2114.566228] Lustre: Skipped 8 previous similar messages [ 2114.572601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.578045] Lustre: Skipped 8 previous similar messages [ 2114.580273] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2114.582627] Lustre: Skipped 8 previous similar messages [ 2114.584115] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2114.588160] LustreError: Skipped 8 previous similar messages [ 2114.590184] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2114.591867] Lustre: Skipped 8 previous similar messages [ 2118.756662] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2118.758488] Lustre: Skipped 16 previous similar messages [ 2118.763490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2118.766937] Lustre: Skipped 16 previous similar messages [ 2118.768348] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2118.771691] Lustre: Skipped 16 previous similar messages [ 2118.773922] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2118.776945] LustreError: Skipped 16 previous similar messages [ 2118.778887] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2118.782201] Lustre: Skipped 16 previous similar messages [ 2126.892707] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2126.896060] Lustre: Skipped 32 previous similar messages [ 2126.901945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2126.904715] Lustre: Skipped 32 previous similar messages [ 2126.905926] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2126.907865] Lustre: Skipped 32 previous similar messages [ 2126.909192] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2126.913223] LustreError: Skipped 32 previous similar messages [ 2126.915308] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2126.917054] Lustre: Skipped 32 previous similar messages [ 2142.940851] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2142.944588] Lustre: Skipped 65 previous similar messages [ 2142.950782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.956775] Lustre: Skipped 65 previous similar messages [ 2142.959149] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2142.960910] Lustre: Skipped 65 previous similar messages [ 2142.962052] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2142.964682] LustreError: Skipped 65 previous similar messages [ 2142.966434] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2142.968604] Lustre: Skipped 65 previous similar messages [ 2150.033096] LustreError: 9054:0:(osp_precreate.c:674:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2150.037943] LustreError: 9054:0:(osp_precreate.c:1378:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2156.726019] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2156.729343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2161.397996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2161.398819] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2161.403959] Lustre: Skipped 1 previous similar message [ 2162.492582] Lustre: server umount lustre-OST0000 complete [ 2166.021935] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2166.025719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.553891] Lustre: server umount lustre-MDT0000 complete [ 2170.553040] LustreError: 23768:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763163 with bad export cookie 2471769941668303500 [ 2170.554100] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2170.558718] LustreError: 23768:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2170.663449] Lustre: server umount lustre-MDT0001 complete [ 2173.578188] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2174.059985] Key type lgssc unregistered [ 2174.123697] LNet: 24564:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2174.125842] LNet: Removed LNI 192.168.202.133@tcp [ 2182.018291] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2182.176855] alg: No test for adler32 (adler32-zlib) [ 2182.929474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2182.992177] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 2183.048775] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2183.050336] LNet: Accept secure, port 988 [ 2184.581490] Key type lgssc registered [ 2184.782284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2186.704449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2186.708852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.832996] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.844369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.619405] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2190.842926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2190.938982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2191.726205] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2192.755002] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2193.333653] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2195.076406] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2195.079280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2195.171170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.357107] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2197.392620] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2198.172817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3187 to 0x280000401:3233) [ 2204.990490] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2205.445408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.450484] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2205.452616] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2205.455411] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 2205.455429] LustreError: 26476:0:(osp_precreate.c:1286:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2206.181934] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2206.183822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.187656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2206.192051] Lustre: Skipped 1 previous similar message [ 2207.978475] Lustre: server umount lustre-OST0000 complete [ 2225.555391] LustreError: 26476:0:(osp_precreate.c:1286:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 2235.182989] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2235.185833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2235.272752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2236.523626] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2237.566570] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2246.272085] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2246.277225] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.133@tcp (at 0@lo) [ 2246.280856] Lustre: Skipped 1 previous similar message [ 2250.128524] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec [ 2250.709705] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b63af000.ost_server_uuid 50 [ 2251.059997] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b63af000.ost_server_uuid in FULL state after 0 sec [ 2256.294037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2256.294540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2256.298665] Lustre: Skipped 2 previous similar messages [ 2256.299881] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2257.818105] Lustre: server umount lustre-OST0000 complete [ 2261.030026] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2261.034133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2261.040432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2261.041928] Lustre: Skipped 1 previous similar message [ 2264.872772] Lustre: server umount lustre-MDT0000 complete [ 2265.886094] LustreError: 29215:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763259 with bad export cookie 4466208420415839288 [ 2265.887122] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2265.893546] LustreError: 29215:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2265.999969] Lustre: server umount lustre-MDT0001 complete [ 2268.936313] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2269.449498] Key type lgssc unregistered [ 2269.520644] LNet: 31821:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2269.523796] LNet: Removed LNI 192.168.202.133@tcp [ 2279.234329] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2279.397235] alg: No test for adler32 (adler32-zlib) [ 2280.149381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2280.210878] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 2280.260177] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2280.261471] LNet: Accept secure, port 988 [ 2281.789446] Key type lgssc registered [ 2282.019084] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2284.278377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2284.282881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2285.412438] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.425212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2286.211645] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2288.420198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2288.528704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2289.289640] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2291.513318] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2291.518489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2291.613244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2292.856851] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2295.104464] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2295.108746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2295.134279] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2295.140217] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2295.179875] Lustre: lustre-OST0001: new disk, initializing [ 2295.182217] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2295.201536] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2296.202172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 2296.467280] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2297.120271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2297.123235] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2297.137818] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2301.252294] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2305.455931] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2318.166516] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2318.167318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2318.175296] Lustre: Skipped 3 previous similar messages [ 2323.174285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2323.175870] Lustre: Skipped 6 previous similar messages [ 2323.190948] Lustre: server umount lustre-MDT0000 complete [ 2324.205019] LustreError: 1686:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763317 with bad export cookie 6901422785973090567 [ 2324.206272] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2324.212035] LustreError: 1686:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2324.321069] Lustre: server umount lustre-MDT0001 complete [ 2335.478780] Lustre: server umount lustre-OST0000 complete [ 2346.604432] Lustre: server umount lustre-OST0001 complete [ 2347.466976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2347.483870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2347.774186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2347.799499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2348.075680] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2348.082341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2348.104239] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2348.108500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2348.423555] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2348.428708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2348.449806] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2348.454012] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2351.510217] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2353.884331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.038968] LustreError: 137-5: mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.053530] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2354.870702] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2357.143502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2358.030348] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2360.351556] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2360.358871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2360.468380] mount.lustre (9982) used greatest stack depth: 9408 bytes left [ 2361.759021] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2363.974639] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2363.978402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2364.025711] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2364.029023] Lustre: Skipped 2 previous similar messages [ 2365.025796] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3297) [ 2365.031238] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2365.332928] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2371.545624] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2372.416796] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2382.053878] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.054626] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2382.063495] Lustre: Skipped 2 previous similar messages [ 2387.062044] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2387.063677] Lustre: Skipped 6 previous similar messages [ 2387.154451] Lustre: server umount mylustre-MDT0000 complete [ 2388.081124] LustreError: 8250:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763381 with bad export cookie 6901422785973093283 [ 2388.081963] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2388.087072] LustreError: 8250:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2388.191493] Lustre: server umount mylustre-MDT0001 complete [ 2399.196557] Lustre: server umount mylustre-OST0000 complete [ 2410.202719] Lustre: server umount mylustre-OST0001 complete [ 2410.972701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2410.996649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2411.258805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2411.283409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2411.558055] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2411.563432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2411.584132] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2411.587303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2411.850840] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2411.853769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2411.875063] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2411.877862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2414.850743] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2417.023675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2417.146537] LustreError: 137-5: tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.162641] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2417.903675] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2419.984705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2420.792853] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2422.840315] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2422.843156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2424.369551] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2426.541141] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2426.546258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2427.591092] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 2427.591634] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3329) [ 2427.732651] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2433.930130] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2434.749773] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2445.109824] LustreError: 11-0: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2445.111699] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.114429] Lustre: Skipped 1 previous similar message [ 2445.115764] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 2447.145061] Lustre: server umount tfs-MDT0000 complete [ 2448.066129] LustreError: 14692:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763441 with bad export cookie 6901422785973095138 [ 2448.067188] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.071496] LustreError: 14692:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2448.165936] Lustre: server umount tfs-MDT0001 complete [ 2459.152664] Lustre: server umount tfs-OST0000 complete [ 2470.157870] Lustre: server umount tfs-OST0001 complete [ 2470.961716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2470.985935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2471.269526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2471.289718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2471.560029] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2471.563699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2471.580840] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2471.584008] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2471.861560] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2471.866768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2471.889488] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2471.892168] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2474.608431] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2476.839147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.962257] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.976020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2476.977695] Lustre: Skipped 3 previous similar messages [ 2477.765804] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2479.900236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2480.712237] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2483.059375] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2483.063376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2484.358063] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2486.556906] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2486.559945] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2487.605810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3361) [ 2487.624625] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 2487.811453] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2493.985230] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2494.811692] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2502.850135] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 11:38:15 (1712763495) [ 2515.029855] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2515.031857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2515.035661] Lustre: Skipped 3 previous similar messages [ 2515.037173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2515.039760] Lustre: Skipped 3 previous similar messages [ 2519.811696] Lustre: server umount lustre-MDT0000 complete [ 2520.661911] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.666407] LustreError: Skipped 3 previous similar messages [ 2520.752089] LustreError: 21052:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763514 with bad export cookie 6901422785973096769 [ 2520.752989] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2520.758830] LustreError: 21052:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2525.669919] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.672661] Lustre: Skipped 4 previous similar messages [ 2526.857809] Lustre: server umount lustre-MDT0001 complete [ 2533.883360] Lustre: server umount lustre-OST0000 complete [ 2541.094781] Lustre: server umount lustre-OST0001 complete [ 2542.484390] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 2544.096424] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2546.148600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2547.706435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2549.146743] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2549.151452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2550.616438] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2550.619979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2552.313515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2552.329897] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 2552.337055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 2552.402830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2552.411597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2552.446760] Lustre: lustre-MDT0000: new disk, initializing [ 2552.464467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2552.466768] Lustre: Skipped 3 previous similar messages [ 2552.470960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2553.261764] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2556.706959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2556.726109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2556.757499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2556.759145] Lustre: Skipped 1 previous similar message [ 2556.796347] Lustre: lustre-MDT0001: new disk, initializing [ 2556.829098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2556.831859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2557.725872] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2561.223203] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2561.227783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2561.248938] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2561.251763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2561.327395] Lustre: lustre-OST0000: new disk, initializing [ 2561.329704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2562.615567] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2562.733299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2562.737993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2562.757824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2566.308839] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2566.313396] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2566.338423] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2566.343447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2566.389295] Lustre: lustre-OST0001: new disk, initializing [ 2566.392452] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2567.422637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2567.425004] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2567.435503] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2567.726665] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2573.186139] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 11:39:25 (1712763565) [ 2577.120077] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 11:39:29 (1712763569) [ 2592.471068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.479293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.483348] Lustre: Skipped 6 previous similar messages [ 2593.914568] Lustre: server umount lustre-OST0000 complete [ 2596.870101] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2597.478512] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.482389] LustreError: Skipped 3 previous similar messages [ 2601.148293] Lustre: server umount lustre-MDT0000 complete [ 2602.224589] LustreError: 28911:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763595 with bad export cookie 6901422785973098288 [ 2602.226172] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2602.232897] LustreError: 28911:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2608.364198] Lustre: server umount lustre-MDT0001 complete [ 2611.356137] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2620.557889] Lustre: server umount lustre-OST0001 complete [ 2621.929065] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 2626.208667] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2628.554776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2630.393466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2631.962614] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2631.965302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2633.438397] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2633.440937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2635.517740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2635.533813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.610660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2635.612957] Lustre: Skipped 1 previous similar message [ 2635.620374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2635.655626] Lustre: lustre-MDT0000: new disk, initializing [ 2635.675195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2636.416314] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2640.557397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2640.577650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2640.728318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2641.718180] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2644.158790] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2644.711139] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2646.564676] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2646.568446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2646.589081] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2646.592107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2647.821841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2648.023045] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2650.412069] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2652.805903] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2652.807958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2652.810720] Lustre: Skipped 5 previous similar messages [ 2652.812317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2652.813612] Lustre: Skipped 6 previous similar messages [ 2658.444823] Lustre: server umount lustre-OST0000 complete [ 2660.742422] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2660.747588] LustreError: Skipped 1 previous similar message [ 2665.708355] Lustre: server umount lustre-MDT0000 complete [ 2666.777658] LustreError: 5745:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763660 with bad export cookie 6901422785973099716 [ 2666.780119] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2666.789996] LustreError: 5745:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2666.903772] Lustre: server umount lustre-MDT0001 complete [ 2670.009786] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2670.544792] Key type lgssc unregistered [ 2670.612717] LNet: 9627:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2670.616236] LNet: Removed LNI 192.168.202.133@tcp [ 2680.164137] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2680.316310] alg: No test for adler32 (adler32-zlib) [ 2681.067408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2681.122372] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 2681.170970] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2681.172257] LNet: Accept secure, port 988 [ 2682.698435] Key type lgssc registered [ 2682.908599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2684.974734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2684.981689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.113621] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.123829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2686.883547] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2689.077806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2689.170021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2690.137631] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2691.196248] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2691.754461] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2693.445501] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2693.449075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2693.559659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2694.840286] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2696.069709] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2696.370555] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 2696.560380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2699.558302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.558410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2702.922749] Lustre: server umount lustre-OST0000 complete [ 2704.181940] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2704.183929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.187136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2704.189230] Lustre: Skipped 1 previous similar message [ 2709.573890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2709.575599] Lustre: Skipped 2 previous similar messages [ 2709.969621] Lustre: server umount lustre-MDT0000 complete [ 2710.953350] LustreError: 11569:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763704 with bad export cookie 17997770197178498251 [ 2710.955483] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2710.962851] LustreError: 11569:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2711.071193] Lustre: server umount lustre-MDT0001 complete [ 2712.935045] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2713.474052] Key type lgssc unregistered [ 2713.544581] LNet: 15269:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2713.546755] LNet: Removed LNI 192.168.202.133@tcp [ 2718.269808] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2718.378022] alg: No test for adler32 (adler32-zlib) [ 2719.128379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2719.184379] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 2719.233211] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2719.235084] LNet: Accept secure, port 988 [ 2720.764470] Key type lgssc registered [ 2720.975613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2723.175494] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2723.178317] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2723.181404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2740.244544] LustreError: 16489:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 1: rc = -5 [ 2776.248439] LustreError: 16489:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 2926.270388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2927.947165] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2929.220333] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2929.888208] Lustre: server umount lustre-OST0000 complete [ 2934.037749] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2934.647277] Key type lgssc unregistered [ 2934.730123] LNet: 18304:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2934.735197] LNet: Removed LNI 192.168.202.133@tcp [ 2948.695088] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2948.848539] alg: No test for adler32 (adler32-zlib) [ 2949.600422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2949.668925] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 2949.712695] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2949.714013] LNet: Accept secure, port 988 [ 2951.238409] Key type lgssc registered [ 2951.447424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2954.971241] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2955.387438] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2955.717156] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 2955.720932] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2956.064218] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 2956.067222] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2956.320006] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2957.428702] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2958.494901] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 2958.498857] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2959.588894] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 2959.591908] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2960.775585] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2960.790750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2960.795916] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2960.818874] Lustre: 23211:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2960.825334] Lustre: 23211:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 2960.832737] Lustre: 23211:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 2960.837969] Lustre: 23211:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 2960.843391] Lustre: 23211:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 2960.848907] Lustre: 23211:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 2960.854739] Lustre: 23211:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 2960.858876] Lustre: 23211:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 2960.862666] Lustre: 23211:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 2960.866504] Lustre: 23211:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 2960.870305] Lustre: 23211:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 2960.874092] Lustre: 23211:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 2960.877799] Lustre: 23211:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 2960.881785] Lustre: 23211:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 2963.304101] Lustre: server umount MGS complete [ 2963.591456] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.721920] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.754072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2963.771716] Lustre: lustre-MDT0000: Aborting client recovery [ 2964.093400] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2964.118738] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2964.134905] Lustre: 23759:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2964.139457] Lustre: 23759:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 2964.142803] Lustre: 23759:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 2964.146225] Lustre: 23759:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 2964.152497] Lustre: 23759:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 2964.155996] Lustre: 23759:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 2964.161523] Lustre: 23759:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 2964.172750] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2964.256050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2964.275445] Lustre: lustre-MDT0001: Aborting client recovery [ 2964.594811] LDISKFS-fs (loop2): file extents enabled, maximum tree depth=5 [ 2964.598722] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2964.615256] LDISKFS-fs (loop2): file extents enabled, maximum tree depth=5 [ 2964.618656] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2964.629796] Lustre: 23852:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2964.633280] Lustre: 23852:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 2964.637515] Lustre: 23852:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 2964.641694] Lustre: 23852:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 2964.646178] Lustre: 23852:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 2964.651002] Lustre: 23852:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 2964.655388] Lustre: 23852:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 2964.699355] Lustre: Found index 0 for lustre-OST0000, updating log [ 2964.736817] Lustre: lustre-OST0000: Aborting recovery [ 2965.016693] LDISKFS-fs (loop3): file extents enabled, maximum tree depth=5 [ 2965.020834] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2965.042107] LDISKFS-fs (loop3): file extents enabled, maximum tree depth=5 [ 2965.045730] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2965.058782] Lustre: 23947:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2965.063473] Lustre: 23947:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 2965.068802] Lustre: 23947:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 2965.073955] Lustre: 23947:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 2965.078883] Lustre: 23947:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 2965.083943] Lustre: 23947:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 2965.088298] Lustre: 23947:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 2970.122902] LustreError: 23690:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -108 [ 2974.118852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 2974.119305] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 2974.121661] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 2974.122281] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 2974.123556] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 2974.134217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 2975.204221] LustreError: 23660:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 220us: rc = -2 [ 2989.141988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.142135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2989.147574] Lustre: Skipped 3 previous similar messages [ 2991.657842] Lustre: server umount lustre-MDT0000 complete [ 2991.925774] LustreError: 23646:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763985 with bad export cookie 14976105951875860695 [ 2991.926928] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2991.936349] LustreError: 23646:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2994.149745] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2994.149858] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2994.149895] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2994.149896] Lustre: Skipped 3 previous similar messages [ 2994.159897] Lustre: Skipped 1 previous similar message [ 2996.153652] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.153654] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2996.153661] Lustre: Skipped 1 previous similar message [ 2996.159279] LustreError: Skipped 2 previous similar messages [ 2998.027921] Lustre: server umount lustre-MDT0001 complete [ 3004.273095] Lustre: server umount lustre-OST0000 complete [ 3010.656512] Lustre: server umount lustre-OST0001 complete [ 3015.904227] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 11:46:48 (1712764008) [ 3026.226110] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3028.966490] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3032.585344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3035.138857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3037.867657] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3037.881175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3040.423757] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3040.430141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3043.216947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3043.242974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3043.344580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3043.360811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3043.405806] Lustre: lustre-MDT0000: new disk, initializing [ 3043.456783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3043.461054] Lustre: Skipped 2 previous similar messages [ 3043.471394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3044.662868] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3049.317142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3049.355088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3049.392483] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3049.423493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3049.427392] Lustre: Skipped 1 previous similar message [ 3049.472317] Lustre: lustre-MDT0001: new disk, initializing [ 3049.526499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3049.546227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3049.551812] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3050.711221] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3053.647416] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3054.510700] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3057.116613] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3057.123719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3057.156184] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3057.166517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3057.281656] Lustre: lustre-OST0000: new disk, initializing [ 3057.285847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3057.319679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.104368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3058.109552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3058.160622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3059.119335] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3061.954605] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3064.488079] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3074.778435] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3083.273946] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3128.245985] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3128.250177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.257527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3128.260399] Lustre: Skipped 1 previous similar message [ 3129.451943] Lustre: server umount lustre-OST0000 complete [ 3133.271299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.272746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3133.272747] Lustre: Skipped 1 previous similar message [ 3133.281411] Lustre: Skipped 2 previous similar messages [ 3136.853763] Lustre: server umount lustre-MDT0000 complete [ 3138.043771] LustreError: 1120:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764131 with bad export cookie 14976105951875863355 [ 3138.044393] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3138.054449] LustreError: 1120:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3138.195818] Lustre: server umount lustre-MDT0001 complete [ 3140.827304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.797904] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3143.312243] Lustre: server umount MGS complete [ 3146.723937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.897611] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.910965] LustreError: 4331:0:(qmt_pool.c:1520:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3146.932855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.843672] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3150.379298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3151.404457] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3152.639430] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3153.322575] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3155.338515] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3155.343926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3156.894218] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3158.097323] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3170.486536] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3170.486623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3170.487420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3170.487421] Lustre: Skipped 1 previous similar message [ 3175.561586] Lustre: server umount lustre-OST0000 complete [ 3180.501984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3180.506972] Lustre: Skipped 2 previous similar messages [ 3182.762275] Lustre: server umount lustre-MDT0000 complete [ 3183.995202] LustreError: 4344:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764177 with bad export cookie 14976105951875865105 [ 3183.996557] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3184.004749] LustreError: 4344:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3184.128692] Lustre: server umount lustre-MDT0001 complete [ 3187.463498] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3188.059908] Key type lgssc unregistered [ 3188.135044] LNet: 8554:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3188.139492] LNet: Removed LNI 192.168.202.133@tcp [ 3200.835414] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3205.418034] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3205.561170] alg: No test for adler32 (adler32-zlib) [ 3206.312416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3206.387435] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3206.448791] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3206.450984] LNet: Accept secure, port 988 [ 3207.985454] Key type lgssc registered [ 3208.269545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3210.851169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3212.816038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3214.579785] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3214.584209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3216.404827] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3216.410854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3218.692414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3218.714075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3218.722884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.817412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3219.832134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3219.868941] Lustre: lustre-MDT0000: new disk, initializing [ 3219.901844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.913976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3220.905125] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3224.989492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3225.014506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3225.043175] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3225.061865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3225.064876] Lustre: Skipped 1 previous similar message [ 3225.104855] Lustre: lustre-MDT0001: new disk, initializing [ 3225.141703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3225.159198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3225.165572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3226.174166] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3228.695462] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3229.365456] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3231.488009] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3231.492078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3231.512634] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3231.517646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3231.593027] Lustre: lustre-OST0000: new disk, initializing [ 3231.595416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3231.622967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3232.142287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3232.146462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3232.177139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3233.053737] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3235.567847] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3238.820732] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3247.574988] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3256.954877] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3266.466762] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3302.278038] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3302.278536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.278805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3307.286148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3307.287606] Lustre: Skipped 2 previous similar messages [ 3307.791470] Lustre: server umount lustre-OST0000 complete [ 3310.277954] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3310.280220] LustreError: Skipped 1 previous similar message [ 3310.281573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3310.285527] Lustre: Skipped 1 previous similar message [ 3310.287422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3312.293937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3312.294047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3312.300766] Lustre: Skipped 1 previous similar message [ 3314.798947] Lustre: server umount lustre-MDT0000 complete [ 3315.864929] LustreError: 13717:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764309 with bad export cookie 7530435436441870416 [ 3315.865531] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3315.874112] LustreError: 13717:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3315.981336] Lustre: server umount lustre-MDT0001 complete [ 3318.551720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.431000] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3320.718410] Lustre: server umount MGS complete [ 3323.801491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.961117] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3323.972007] LustreError: 21042:0:(qmt_pool.c:1520:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3323.989243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3324.764232] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3327.041852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3327.914461] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3328.914686] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3329.493949] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3331.193022] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3331.196309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3331.287226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3331.290061] Lustre: Skipped 1 previous similar message [ 3332.564657] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3333.570048] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3357.317823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3357.317916] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3357.318103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3357.323650] Lustre: Skipped 1 previous similar message [ 3360.559367] Lustre: server umount lustre-OST0000 complete [ 3362.182056] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3362.185006] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3367.334243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3367.336760] Lustre: Skipped 4 previous similar messages [ 3367.655125] Lustre: server umount lustre-MDT0000 complete [ 3368.746182] LustreError: 21010:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764362 with bad export cookie 7530435436441872257 [ 3368.747365] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3368.754475] LustreError: 21010:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3368.854813] Lustre: server umount lustre-MDT0001 complete [ 3371.841443] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3372.299537] Key type lgssc unregistered [ 3372.371652] LNet: 25267:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3372.374022] LNet: Removed LNI 192.168.202.133@tcp [ 3381.601591] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3381.774195] alg: No test for adler32 (adler32-zlib) [ 3382.525345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3382.590794] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3382.641140] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3382.643707] LNet: Accept secure, port 988 [ 3384.175437] Key type lgssc registered [ 3384.390891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3386.385880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3386.389990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3387.505314] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.516385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.280058] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3390.402767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3390.495273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3391.174283] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3392.189188] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3392.767486] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3394.427194] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3394.431589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3394.517353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3395.709196] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3396.703865] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3398.366109] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3398.368659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3398.383206] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3398.386070] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3398.419348] Lustre: lustre-OST0001: new disk, initializing [ 3398.421510] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3398.442444] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3399.653568] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3403.469909] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3403.522624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3404.095608] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012ab5f800.ost_server_uuid 50 [ 3404.440677] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab5f800.ost_server_uuid in FULL state after 0 sec [ 3404.988352] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012ab5f800.ost_server_uuid 50 [ 3405.288853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3405.293181] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3405.301096] LustreError: 31343:0:(osp_precreate.c:992:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 3405.303107] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88012ab5f800.ost_server_uuid in FULL state after 0 sec [ 3405.309327] LustreError: 31343:0:(osp_precreate.c:992:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 3406.308782] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3417.711693] Lustre: lustre-OST0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 3420.326434] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.327292] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3420.332438] Lustre: Skipped 1 previous similar message [ 3422.431854] Lustre: server umount lustre-OST0001 complete [ 3422.711038] LustreError: 137-5: lustre-OST0001: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3425.334115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3425.334120] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3425.334831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3425.334832] Lustre: Skipped 1 previous similar message [ 3425.348029] Lustre: Skipped 1 previous similar message [ 3429.544713] Lustre: server umount lustre-OST0000 complete [ 3430.533798] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3430.536225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3430.539489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3430.541760] Lustre: Skipped 1 previous similar message [ 3435.349838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3435.350042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3435.357494] Lustre: Skipped 1 previous similar message [ 3436.584717] Lustre: server umount lustre-MDT0000 complete [ 3437.595035] LustreError: 27278:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764430 with bad export cookie 9140834655502800393 [ 3437.596184] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3437.604928] LustreError: 27278:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3437.710390] Lustre: server umount lustre-MDT0001 complete [ 3440.693318] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3441.218752] Key type lgssc unregistered [ 3441.290584] LNet: 498:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3441.293357] LNet: Removed LNI 192.168.202.133@tcp [ 3449.731057] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3449.895001] alg: No test for adler32 (adler32-zlib) [ 3450.646444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3450.700241] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3450.745838] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3450.747791] LNet: Accept secure, port 988 [ 3452.273384] Key type lgssc registered [ 3452.466753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3454.207471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3454.212867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.322296] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3455.342842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3456.024633] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3456.937636] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3458.486456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3458.582505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3459.256755] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3461.449824] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3461.452962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3461.530101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.644281] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3463.548898] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3465.093865] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3465.097414] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3465.141842] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3466.141928] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3466.148045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:68 to 0x280000401:97) [ 3466.297948] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3467.245686] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3469.093465] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3469.144061] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3543.269837] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3543.270433] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3543.271095] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3548.196740] Lustre: server umount lustre-OST0001 complete [ 3548.278506] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3548.282591] LustreError: Skipped 1 previous similar message [ 3551.285825] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3551.288447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.292418] Lustre: Skipped 1 previous similar message [ 3551.294191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.296525] Lustre: Skipped 1 previous similar message [ 3553.286081] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3553.286088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3553.286635] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3553.286646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.300031] LustreError: Skipped 1 previous similar message [ 3555.377250] Lustre: server umount lustre-OST0000 complete [ 3558.293834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3558.297239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3558.298823] Lustre: Skipped 2 previous similar messages [ 3562.340136] Lustre: server umount lustre-MDT0000 complete [ 3563.293375] LustreError: 2412:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764556 with bad export cookie 4178857026075360356 [ 3563.296157] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3563.302637] LustreError: 2412:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3563.402004] Lustre: server umount lustre-MDT0001 complete [ 3566.203107] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3566.658302] Key type lgssc unregistered [ 3566.717773] LNet: 7753:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3566.720009] LNet: Removed LNI 192.168.202.133@tcp [ 3581.784742] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3581.979091] alg: No test for adler32 (adler32-zlib) [ 3582.731390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3582.792797] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3582.844955] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3582.847592] LNet: Accept secure, port 988 [ 3584.375451] Key type lgssc registered [ 3584.589288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3586.925322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3586.932168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3588.066655] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3588.085450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3588.952103] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3591.148061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3591.249811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3591.996830] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3594.116553] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3594.119685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3594.204823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3595.392022] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3597.523665] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3597.527550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3597.578034] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3598.577302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:68 to 0x280000401:129) [ 3598.595034] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 3598.748936] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3605.888793] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3607.955121] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 3616.261933] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3616.264813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3616.270089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3621.606405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3621.608540] Lustre: Skipped 6 previous similar messages [ 3621.623810] Lustre: server umount lustre-MDT0000 complete [ 3622.650020] LustreError: 10785:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764615 with bad export cookie 7690847093628969033 [ 3622.651624] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3622.658471] LustreError: 10785:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3622.765913] Lustre: server umount lustre-MDT0001 complete [ 3633.863519] Lustre: server umount lustre-OST0000 complete [ 3644.949671] Lustre: server umount lustre-OST0001 complete [ 3648.400731] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3650.691874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3650.805455] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3650.820145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3651.585164] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3653.739765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3654.583436] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3656.738094] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3656.741656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3658.024017] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3660.178058] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3660.182140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3660.231939] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3660.234954] Lustre: Skipped 2 previous similar messages [ 3661.233305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:68 to 0x280000401:161) [ 3661.253649] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 3661.444489] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3667.691754] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3668.585907] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 3678.853871] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3678.857691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3678.863375] Lustre: Skipped 3 previous similar messages [ 3678.865895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3680.334547] Lustre: server umount lustre-MDT0000 complete [ 3681.314985] LustreError: 17126:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764674 with bad export cookie 7690847093628970657 [ 3681.316417] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3681.322337] LustreError: 17126:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3681.417867] Lustre: server umount lustre-MDT0001 complete [ 3692.452162] Lustre: server umount lustre-OST0000 complete [ 3703.464750] Lustre: server umount lustre-OST0001 complete [ 3704.824881] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 3705.190468] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 3706.585194] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 11:58:19 (1712764699) [ 3711.238643] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3713.311109] SGI XFS with ACLs, security attributes, no debug enabled [ 3713.315894] XFS (loop0): Mounting V5 Filesystem [ 3713.324015] XFS (loop0): Ending clean mount [ 3715.644867] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [ 3715.766841] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3719.771684] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3721.285507] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3723.511167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3725.133004] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3726.610856] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3726.616079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3728.129484] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3728.132411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3729.824507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3729.839115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3729.907731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3729.916037] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3729.950294] Lustre: lustre-MDT0000: new disk, initializing [ 3729.968605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3729.974446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3730.734507] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3734.172559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3734.192292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3734.220524] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3734.232814] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3734.234673] Lustre: Skipped 1 previous similar message [ 3734.268837] Lustre: lustre-MDT0001: new disk, initializing [ 3734.296853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3734.300208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3735.083368] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3737.324419] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3737.877991] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3739.570191] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3739.573824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3739.588529] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3739.591037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3739.659962] Lustre: lustre-OST0000: new disk, initializing [ 3739.661959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3740.668341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3740.671054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3740.687052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3740.873995] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3743.127339] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3743.912341] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3743.959611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3744.737825] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3744.780938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3745.685750] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3745.686352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3745.686354] Lustre: Skipped 3 previous similar messages [ 3745.686932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3745.686934] Lustre: Skipped 4 previous similar messages [ 3745.694661] LustreError: Skipped 1 previous similar message [ 3750.694353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3750.697436] Lustre: Skipped 2 previous similar messages [ 3751.321468] Lustre: server umount lustre-OST0000 complete [ 3754.309903] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3754.313410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.318287] Lustre: Skipped 1 previous similar message [ 3758.337400] Lustre: server umount lustre-MDT0000 complete [ 3759.301066] LustreError: 30674:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764752 with bad export cookie 7690847093628972176 [ 3759.303548] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3759.307554] LustreError: 30674:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3759.407276] Lustre: server umount lustre-MDT0001 complete [ 3761.452656] XFS (loop0): Unmounting Filesystem [ 3764.225434] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 11:59:16 (1712764756) [ 3765.972069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3766.098463] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3766.120253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3766.123043] Lustre: Skipped 2 previous similar messages [ 3766.878753] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3769.009659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3769.877104] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3770.889881] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3771.429389] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3773.126358] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3773.130324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3774.420500] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3775.411419] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3784.230009] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3784.230103] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3784.230816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3784.230817] Lustre: Skipped 3 previous similar messages [ 3784.241048] Lustre: Skipped 2 previous similar messages [ 3784.867323] Lustre: server umount lustre-OST0000 complete [ 3791.890428] Lustre: server umount lustre-MDT0000 complete [ 3792.834117] LustreError: 32657:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764786 with bad export cookie 7690847093628973030 [ 3792.835554] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3792.841286] LustreError: 32657:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3792.941088] Lustre: server umount lustre-MDT0001 complete [ 3795.818488] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3796.291238] Key type lgssc unregistered [ 3796.348636] LNet: 4006:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3796.351380] LNet: Removed LNI 192.168.202.133@tcp [ 3804.998496] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3805.171352] alg: No test for adler32 (adler32-zlib) [ 3805.922360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3805.976608] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3806.023131] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3806.025015] LNet: Accept secure, port 988 [ 3807.552423] Key type lgssc registered [ 3807.756179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3809.572048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3809.577035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3810.701893] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.713784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3811.469955] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3813.633895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3813.724172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3814.495793] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3815.542669] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3816.116757] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3817.828982] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3817.835070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3817.925703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3819.114298] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3820.135586] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3828.933989] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3828.938265] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3828.940878] Lustre: Skipped 1 previous similar message [ 3831.260682] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 3833.941963] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3834.306320] Lustre: server umount lustre-MDT0001 complete [ 3835.062763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3836.262956] LustreError: 137-5: lustre-MDT0001: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3836.987043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3837.012641] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3837.014083] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3837.024594] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3837.093227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3837.841308] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3842.103440] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 3844.847519] LustreError: 10221:0:(obd_mount.c:210:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 3844.850389] LustreError: 10221:0:(tgt_mount.c:683:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 3844.853442] LustreError: 6095:0:(mgc_request.c:621:do_requeue()) failed processing log: -17 [ 3844.864919] LustreError: 10224:0:(obd_config.c:2037:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -17 [ 3844.868698] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 4141.757409] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 4141.759239] Lustre: Skipped 1 previous similar message [ 4142.582156] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4142.582398] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4142.586758] Lustre: Skipped 2 previous similar messages [ 4146.533388] Lustre: server umount lustre-MDT0001 complete [ 4146.759042] LustreError: 137-5: lustre-MDT0001: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.255960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4149.182293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4149.209551] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4149.211054] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4149.220174] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4149.291512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4150.074093] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4154.295406] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4157.394675] LustreError: 27876:0:(obd_mount.c:210:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4157.397063] LustreError: 27876:0:(tgt_mount.c:683:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4157.399574] LustreError: 6095:0:(mgc_request.c:621:do_requeue()) failed processing log: -17 [ 4157.401483] LustreError: 6095:0:(mgc_request.c:621:do_requeue()) Skipped 1 previous similar message [ 4157.413361] LustreError: 27890:0:(obd_config.c:703:class_setup()) Device 8 already setup (type osp) [ 4157.415244] LustreError: 27890:0:(obd_config.c:2037:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -17 [ 4157.418369] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.133@tcp [ 4452.257684] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 4452.261884] Lustre: Skipped 2 previous similar messages [ 4454.774046] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4454.775645] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.959502] Lustre: server umount lustre-MDT0001 complete [ 4459.008349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4459.782134] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4459.786023] LustreError: Skipped 3 previous similar messages [ 4461.017961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4461.044864] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4461.046791] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4461.058076] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4461.140768] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4461.922994] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4466.151827] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4468.905821] LustreError: 13081:0:(obd_mount.c:210:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4468.908247] LustreError: 13081:0:(tgt_mount.c:683:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4468.910904] LustreError: 6095:0:(mgc_request.c:621:do_requeue()) failed processing log: -17 [ 4468.914378] LustreError: 6095:0:(mgc_request.c:621:do_requeue()) Skipped 1 previous similar message [ 4468.927193] LustreError: 13084:0:(lod_lov.c:231:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 4468.933584] LustreError: 13084:0:(obd_config.c:2037:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -114 [ 4468.936469] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 4776.613902] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4776.616144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.619366] Lustre: Skipped 1 previous similar message [ 4776.620975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4776.624156] Lustre: Skipped 3 previous similar messages [ 4781.041818] Lustre: server umount lustre-MDT0000 complete [ 4781.653727] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4781.657723] LustreError: Skipped 2 previous similar messages [ 4781.996027] LustreError: 11591:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765775 with bad export cookie 2850819576622327181 [ 4781.997607] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4782.004354] LustreError: 11591:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4786.662027] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4786.665754] Lustre: Skipped 2 previous similar messages [ 4788.113896] Lustre: server umount lustre-MDT0001 complete [ 4789.177152] Lustre: server umount lustre-OST0000 complete [ 4791.191088] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 4792.743128] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4794.929190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4796.578917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4798.102419] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4798.106096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4799.660280] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4799.664801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4802.771421] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 12:16:35 (1712765795) [ 4804.461682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4804.481700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4804.554531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4804.562620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4804.598216] Lustre: lustre-MDT0000: new disk, initializing [ 4804.616109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4804.624663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4805.364019] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4808.708993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4808.727630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4808.756795] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4808.770781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4808.772434] Lustre: Skipped 1 previous similar message [ 4808.809528] Lustre: lustre-MDT0001: new disk, initializing [ 4808.832946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4808.842328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4808.845617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4809.588217] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4811.805122] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4812.364862] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4814.059331] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4814.062588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4814.080677] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4814.083853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4814.163067] Lustre: lustre-OST0000: new disk, initializing [ 4814.165641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4814.185566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4815.374913] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4817.632235] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4819.186322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4819.190274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4819.206113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4828.854032] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4828.856472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4828.859605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4828.861323] Lustre: Skipped 4 previous similar messages [ 4830.773043] Lustre: server umount lustre-MDT0000 complete [ 4831.767103] LustreError: 888:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765825 with bad export cookie 2850819576622329078 [ 4831.768110] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4831.768593] LustreError: 1861:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 4831.768636] LustreError: 1861:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 4831.768640] LustreError: 1861:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 4831.780697] LustreError: 888:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4831.876411] Lustre: server umount lustre-MDT0001 complete [ 4843.042956] Lustre: server umount lustre-OST0000 complete [ 4849.561736] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:17:21 (1712765841) [ 4857.727815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4857.955720] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4857.963211] LustreError: Skipped 2 previous similar messages [ 4857.993643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4859.301391] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4863.086150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4864.262788] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4865.758725] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4866.767748] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4867.872347] Lustre: Failing over lustre-MDT0000 [ 4867.882754] LustreError: 6713:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4867.899590] LustreError: 6713:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4867.905291] LustreError: 6713:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 4868.007706] Lustre: server umount lustre-MDT0000 complete [ 4868.230295] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4868.234584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4868.239424] Lustre: Skipped 2 previous similar messages [ 4878.280888] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4878.285879] LustreError: Skipped 7 previous similar messages [ 4880.910962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4880.967095] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4881.052136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4881.054933] Lustre: Skipped 1 previous similar message [ 4881.068670] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4881.851154] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4884.272933] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 4886.054726] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4886.057674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4886.067406] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4888.063849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4888.067379] Lustre: Skipped 4 previous similar messages [ 4892.403932] Lustre: server umount lustre-MDT0000 complete [ 4894.572848] LustreError: 6669:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765887 with bad export cookie 2850819576622330947 [ 4894.579417] LustreError: 6669:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4894.581237] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4894.775030] Lustre: server umount lustre-MDT0001 complete [ 4895.283398] hrtimer: interrupt took 7128938 ns [ 4899.762970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4900.157909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4902.714985] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4909.498380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4912.597419] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4915.886078] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4917.925582] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4919.337318] Lustre: Failing over lustre-MDT0000 [ 4919.353609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4919.358642] Lustre: Skipped 3 previous similar messages [ 4919.480557] Lustre: server umount lustre-MDT0000 complete [ 4920.382173] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4920.388292] LustreError: Skipped 5 previous similar messages [ 4934.634597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4934.715450] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4934.877093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4934.879667] Lustre: Skipped 1 previous similar message [ 4934.905439] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4935.874326] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4935.880938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4935.888244] Lustre: Skipped 1 previous similar message [ 4935.894200] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4936.551989] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4939.848189] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 4948.325803] Lustre: server umount lustre-MDT0000 complete [ 4949.988122] LustreError: 10203:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765943 with bad export cookie 2850819576622331864 [ 4949.990300] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4950.008028] LustreError: 10203:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4950.165676] Lustre: server umount lustre-MDT0001 complete [ 4956.676717] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:19:08 (1712765948) [ 4964.829157] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 4969.659158] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4975.160377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4979.335994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4982.951829] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4982.962782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4986.773695] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4986.788961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4991.697754] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4997.756622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4997.810737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4997.952007] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4997.973776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4998.022052] Lustre: lustre-MDT0000: new disk, initializing [ 4998.071064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4999.712946] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5005.127223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5005.160362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5005.217777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5005.220376] Lustre: Skipped 1 previous similar message [ 5005.269615] Lustre: lustre-MDT0001: new disk, initializing [ 5005.312451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5005.315075] Lustre: Skipped 1 previous similar message [ 5005.331609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5005.337029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5006.692211] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5009.397831] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5010.239643] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5012.524965] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5012.528806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5012.546675] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5012.552447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5014.168290] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5016.550735] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5019.094788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5019.097344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5019.106631] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 5019.114321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5019.118989] LustreError: 20436:0:(ofd_dev.c:1597:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x280000401:33, last_id 65281. Likely MDS last_id corruption [ 5019.124448] LustreError: 20435:0:(osp_precreate.c:674:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -22 [ 5019.127109] LustreError: 20435:0:(osp_precreate.c:1378:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -22 [ 5023.319053] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 5029.126181] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_create to node 0@lo failed: rc = -107 [ 5029.126656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5029.126657] Lustre: Skipped 3 previous similar messages [ 5029.126943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5029.126944] Lustre: Skipped 4 previous similar messages [ 5029.139083] LustreError: Skipped 2 previous similar messages [ 5032.546240] Lustre: server umount lustre-OST0000 complete [ 5033.823295] LustreError: 20435:0:(osp_precreate.c:674:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5033.828452] LustreError: 20435:0:(osp_precreate.c:1378:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5039.917879] Lustre: server umount lustre-MDT0000 complete [ 5041.587363] LustreError: 17738:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712766034 with bad export cookie 2850819576622332249 [ 5041.588906] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5041.606962] LustreError: 17738:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5041.807304] Lustre: server umount lustre-MDT0001 complete [ 5046.779960] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5047.848316] Key type lgssc unregistered [ 5047.938134] LNet: 21880:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5047.942094] LNet: Removed LNI 192.168.202.133@tcp [ 5059.023924] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5059.214953] alg: No test for adler32 (adler32-zlib) [ 5059.966400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5060.057659] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 5060.133236] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5060.135509] LNet: Accept secure, port 988 [ 5061.689409] Key type lgssc registered [ 5061.990733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5064.329078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5064.334904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5065.487851] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5065.502361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5066.690340] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5069.702123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5069.872736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5070.949927] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5072.797572] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5073.726720] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5076.726220] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5076.737984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5076.901195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5078.531982] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5079.899529] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 5079.988873] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5100.750701] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:21:32 (1712766092) [ 5101.960559] Lustre: Modifying parameter general.jobid_name in log params [ 5103.080623] Lustre: Modifying parameter general.jobid_name in log params [ 5113.872082] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:21:45 (1712766105) [ 5124.569600] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 12:21:56 (1712766116) [ 5125.563158] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb in log lustre-client [ 5133.468103] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:22:05 (1712766125) [ 5134.503077] Lustre: Setting parameter general.osc.*.max_pages_per_rpc in log params [ 5134.521723] Lustre: Skipped 2 previous similar messages [ 5138.642199] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc in log lustre-client [ 5138.646572] Lustre: Skipped 1 previous similar message [ 5146.661353] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc in log params [ 5146.665154] Lustre: Skipped 8 previous similar messages [ 5151.567142] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:22:23 (1712766143) [ 5153.174795] Lustre: *** cfs_fail_loc=131b, val=2*** [ 5155.628216] Lustre: *** cfs_fail_loc=131b, val=2*** [ 5165.417205] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:22:37 (1712766157) [ 5166.767026] Lustre: Modifying parameter general.jobid_name in log params [ 5178.957986] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:22:50 (1712766170) [ 5190.102645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.108723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5191.779803] Lustre: server umount lustre-OST0000 complete [ 5195.110286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5195.121444] Lustre: Skipped 1 previous similar message [ 5195.127566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5195.130880] Lustre: Skipped 2 previous similar messages [ 5199.828945] Lustre: server umount lustre-MDT0000 complete [ 5200.122920] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5200.130906] LustreError: Skipped 1 previous similar message [ 5201.439310] LustreError: 31090:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712766194 with bad export cookie 13354204204824689865 [ 5201.440987] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5201.451013] LustreError: 31090:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5201.656191] Lustre: server umount lustre-MDT0001 complete [ 5206.439473] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5207.739853] Key type lgssc unregistered [ 5207.864936] LNet: 3401:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5207.872292] LNet: Removed LNI 192.168.202.133@tcp [ 5215.971203] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 5222.794748] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5223.006193] alg: No test for adler32 (adler32-zlib) [ 5223.759428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5223.945481] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 5224.080339] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5224.084402] LNet: Accept secure, port 988 [ 5225.648938] Key type lgssc registered [ 5226.010667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5230.225225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5233.491632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5236.553137] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5236.561126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5239.361757] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5239.371130] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5242.341793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5242.379134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5242.398835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5243.513195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5243.536021] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5243.589467] Lustre: lustre-MDT0000: new disk, initializing [ 5243.630136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5243.638798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5244.970113] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5250.078550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5250.108905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5250.141516] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5250.160671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5250.163040] Lustre: Skipped 1 previous similar message [ 5250.202500] Lustre: lustre-MDT0001: new disk, initializing [ 5250.248877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5250.272028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5250.280127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5251.554709] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5254.387736] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5255.447235] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5258.102190] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5258.109087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5258.150574] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5258.160747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5258.303135] Lustre: lustre-OST0000: new disk, initializing [ 5258.305925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5258.338525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5260.122004] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5261.069943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5261.074969] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5261.105608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5262.985833] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5263.991752] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5264.064237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5265.149540] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5265.213778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5266.104230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5266.104268] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5266.104555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5266.121545] Lustre: Skipped 1 previous similar message [ 5271.110284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5271.112396] Lustre: Skipped 2 previous similar messages [ 5271.919126] Lustre: server umount lustre-OST0000 complete [ 5275.270469] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5275.273526] LustreError: Skipped 1 previous similar message [ 5275.275731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.282052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5279.009901] Lustre: server umount lustre-MDT0000 complete [ 5280.150930] LustreError: 7138:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712766273 with bad export cookie 9583471224999208300 [ 5280.151996] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5280.159168] LustreError: 7138:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5280.258219] Lustre: server umount lustre-MDT0001 complete [ 5284.753493] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:24:37 (1712766277) [ 5286.874694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5287.037632] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5287.068613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5287.998343] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5290.623985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5291.711933] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5293.424603] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5294.307822] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5296.682542] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5296.692957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5296.767215] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5296.831551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5296.835685] Lustre: Skipped 1 previous similar message [ 5298.735643] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5300.835971] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5304.113699] LustreError: 14825:0:(llog.c:436:llog_init_handle()) llog type is not specified! [ 5304.599815] Lustre: Modifying parameter general.timeout in log params [ 5306.652756] Lustre: Modifying parameter general.timeout in log params [ 5306.655368] Lustre: Skipped 4 previous similar messages [ 5310.882065] Lustre: Modifying parameter general.timeout in log params [ 5310.884276] Lustre: Skipped 11 previous similar messages [ 5319.326245] Lustre: Modifying parameter general.timeout in log params [ 5319.329527] Lustre: Skipped 19 previous similar messages [ 5335.465547] Lustre: Modifying parameter general.timeout in log params [ 5335.468092] Lustre: Skipped 32 previous similar messages [ 5371.492351] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:26:03 (1712766363) [ 5373.886257] Lustre: Setting parameter general.jobid_var in log params [ 5373.889053] Lustre: Skipped 59 previous similar messages [ 5378.998296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5379.000413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5379.000414] Lustre: Skipped 2 previous similar messages [ 5379.010612] Lustre: Skipped 3 previous similar messages [ 5380.137793] Lustre: server umount lustre-MDT0000 complete [ 5382.304741] LustreError: 15613:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712766375 with bad export cookie 9583471224999209140 [ 5382.305467] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5382.314097] LustreError: 15613:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5384.008041] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5384.008431] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5384.008773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5384.008775] Lustre: Skipped 2 previous similar messages [ 5384.499380] Lustre: server umount lustre-MDT0001 complete [ 5387.086447] Lustre: server umount lustre-OST0000 complete [ 5391.948483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5394.041668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5396.358206] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5396.375383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5398.487060] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5398.495137] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5402.458296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5402.702677] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5402.876936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5404.534141] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5408.990373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5409.044747] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5410.544896] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5412.459240] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5413.513097] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5416.782884] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5416.787359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5416.886664] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5419.160148] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5420.992470] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5426.029352] Lustre: server umount lustre-OST0000 complete [ 5428.665518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5428.666711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5428.674088] Lustre: Skipped 1 previous similar message [ 5437.937125] Lustre: server umount lustre-MDT0000 complete [ 5438.678192] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.693582] LustreError: Skipped 1 previous similar message [ 5440.092793] LustreError: 27385:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712766433 with bad export cookie 9583471224999211345 [ 5440.093318] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5440.105177] LustreError: 27385:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5450.390112] Lustre: server umount lustre-MDT0001 complete [ 5457.063023] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5458.643248] Key type lgssc unregistered [ 5458.783196] LNet: 31330:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5458.789817] LNet: Removed LNI 192.168.202.133@tcp [ 5472.675417] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5472.914335] alg: No test for adler32 (adler32-zlib) [ 5473.679370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5473.805947] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 5473.906796] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5473.910025] LNet: Accept secure, port 988 [ 5475.464471] Key type lgssc registered [ 5475.883341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5478.982883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5478.991903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5480.176695] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5480.211833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5481.269847] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5483.982086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5484.148337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5485.122528] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5486.348846] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5487.079063] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5489.294559] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5489.300104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5489.467025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5490.836565] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5491.989246] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5499.122380] Lustre: Modifying parameter general.jobid_var in log params [ 5504.119199] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5504.123219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5504.130689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5507.192223] Lustre: server umount lustre-MDT0000 complete [ 5508.475151] LustreError: 3987:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712766501 with bad export cookie 1373566902022156483 [ 5508.477026] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5508.490384] LustreError: 3987:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5508.645964] Lustre: server umount lustre-MDT0001 complete [ 5520.077557] Lustre: server umount lustre-OST0000 complete [ 5524.503946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5526.488566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5528.035084] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5528.041771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5529.310909] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5529.316489] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5531.238508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5531.346344] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5531.358799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5531.430334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5532.193800] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5534.975835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5535.023433] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5535.032823] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5536.173479] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5538.006566] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5539.153435] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5542.269878] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5542.277221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5542.387304] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5542.571464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5542.574051] Lustre: Skipped 1 previous similar message [ 5545.168937] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5547.393216] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5548.118065] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5554.583072] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5554.591345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5554.607440] Lustre: Skipped 2 previous similar messages [ 5554.611443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5554.615295] Lustre: Skipped 2 previous similar messages [ 5557.539922] Lustre: server umount lustre-OST0000 complete [ 5559.638119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.641532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5559.641534] Lustre: Skipped 1 previous similar message [ 5559.647952] Lustre: Skipped 2 previous similar messages [ 5564.650511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5564.667942] Lustre: Skipped 2 previous similar messages [ 5565.365365] Lustre: server umount lustre-MDT0000 complete [ 5567.841319] LustreError: 7313:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712766561 with bad export cookie 1373566902022157295 [ 5567.845022] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5568.272532] Lustre: server umount lustre-MDT0001 complete [ 5574.600342] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5576.012883] Key type lgssc unregistered [ 5576.126391] LNet: 10385:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5576.132220] LNet: Removed LNI 192.168.202.133@tcp [ 5621.725067] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5623.319771] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 5625.164152] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules [ 5625.829507] alg: No test for adler32 (adler32-zlib) [ 5626.607511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5626.838385] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 5626.975538] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5626.978723] LNet: Accept secure, port 988 [ 5628.596871] Key type lgssc registered [ 5629.127235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5629.652529] LustreError: 16489:0:(lu_object.c:1416:lu_context_key_register()) cfs_race id 60d sleeping [ 5638.873084] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5643.433421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5651.227839] LustreError: 16489:0:(lu_object.c:1416:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 5651.275138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5652.666312] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5652.726602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5654.262548] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5662.598816] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 12:30:54 (1712766654) [ 5662.741698] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5665.308874] LustreError: 17646:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5665.315325] LustreError: 17646:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5665.325793] LustreError: 17646:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 5665.688325] Lustre: server umount lustre-MDT0000 complete [ 5671.847715] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5672.886234] Key type lgssc unregistered [ 5672.995042] LNet: 19726:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5673.000852] LNet: Removed LNI 192.168.202.133@tcp [ 5680.347046] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5680.554335] alg: No test for adler32 (adler32-zlib) [ 5681.307359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5681.404865] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 5681.496205] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5681.500100] LNet: Accept secure, port 988 [ 5683.049423] Key type lgssc registered [ 5683.595563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5687.306957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5687.317785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.539230] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5688.570495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.944540] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5694.331685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5694.653080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5695.964136] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5697.529867] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5698.606814] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5701.597153] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5701.603119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5701.869669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5704.062205] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5706.012206] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5744.635094] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 12:32:16 (1712766736) [ 5745.343882] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 5748.521865] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 12:32:20 (1712766740) [ 5752.919202] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.919595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5752.924868] Lustre: Skipped 2 previous similar messages [ 5755.789354] Lustre: server umount lustre-MDT0000 complete [ 5757.596714] LustreError: 20962:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712766750 with bad export cookie 12827826358074972291 [ 5757.612688] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5757.757944] Lustre: server umount lustre-MDT0001 complete [ 5770.100896] Lustre: server umount lustre-OST0000 complete [ 5775.481905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5775.720072] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.808552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5777.520965] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5780.810031] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5781.627952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5781.853541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5783.067509] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5784.687229] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5785.485510] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5787.418136] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5787.424575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5789.760005] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5789.766911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5789.792691] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5789.799645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5789.872612] LustreError: 140-5: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 5789.876110] LustreError: 26228:0:(mgs_handler.c:515:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 5789.883526] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 5789.886891] LustreError: 28420:0:(tgt_mount.c:2223:server_fill_super()) Unable to start targets: -98 [ 5789.890468] LustreError: 28420:0:(tgt_mount.c:1755:server_put_super()) no obd lustre-OST0000 [ 5789.895426] LustreError: 28420:0:(tgt_mount.c:135:server_deregister_mount()) lustre-OST0000 not registered [ 5789.962497] Lustre: server umount lustre-OST0000 complete [ 5789.964614] LustreError: 28420:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 5793.332197] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5793.339577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5793.425490] LustreError: 140-5: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 5793.429507] LustreError: 26229:0:(mgs_handler.c:515:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 5793.438701] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 5793.443627] LustreError: 29032:0:(tgt_mount.c:2223:server_fill_super()) Unable to start targets: -98 [ 5793.446578] LustreError: 29032:0:(tgt_mount.c:1755:server_put_super()) no obd lustre-OST0000 [ 5793.449315] LustreError: 29032:0:(tgt_mount.c:135:server_deregister_mount()) lustre-OST0000 not registered [ 5793.502107] Lustre: server umount lustre-OST0000 complete [ 5793.504246] LustreError: 29032:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 5794.523388] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5794.532323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5796.902147] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5796.907794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5796.985749] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5796.988598] Lustre: Found index 0 for lustre-OST0000, updating log [ 5796.991473] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 5797.013835] Lustre: lustre-OST0000: new disk, initializing [ 5797.017545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5797.079705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5798.661806] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5799.894786] Lustre: Failing over lustre-OST0000 [ 5799.955377] Lustre: server umount lustre-OST0000 complete [ 5801.798084] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5801.801407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5801.806176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5801.812340] Lustre: Skipped 2 previous similar messages [ 5806.086152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5806.086428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5806.093894] Lustre: Skipped 1 previous similar message [ 5807.300399] Lustre: server umount lustre-MDT0000 complete [ 5808.628050] LustreError: 26219:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712766801 with bad export cookie 12827826358074975042 [ 5808.629624] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5808.640412] LustreError: 26219:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5808.767175] Lustre: server umount lustre-MDT0001 complete [ 5812.982813] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 12:33:25 (1712766805) [ 5816.585920] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5819.481218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5819.677304] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5819.706834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5820.817336] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5823.383085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5824.630181] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5827.106555] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5827.111498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5828.491008] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5830.686833] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5830.690150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5830.713893] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5830.717890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5830.735244] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5830.754103] Lustre: lustre-OST0001: new disk, initializing [ 5830.756472] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5832.168978] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5833.101222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5833.104747] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5833.121729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5838.438187] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5840.593817] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 5847.285858] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5847.289656] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.295312] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5847.298278] Lustre: Skipped 2 previous similar messages [ 5852.117846] Lustre: server umount lustre-MDT0001 complete [ 5852.839640] LustreError: 137-5: lustre-MDT0001: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5852.930391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5854.648803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5854.670178] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5854.671727] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5854.681540] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5854.810692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5854.812982] Lustre: Skipped 3 previous similar messages [ 5855.569855] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5856.218334] LustreError: 5426:0:(obd_mount.c:210:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 5856.220976] LustreError: 5426:0:(tgt_mount.c:683:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 5856.223666] LustreError: 476:0:(mgc_request.c:621:do_requeue()) failed processing log: -17 [ 5856.240155] LustreError: 5431:0:(obd_config.c:2037:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -17 [ 5856.243506] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5859.492791] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 12:34:11 (1712766851) [ 5859.815454] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 5862.612102] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5872.485172] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5873.386337] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5889.797659] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5889.800244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5889.803926] Lustre: Skipped 2 previous similar messages [ 5889.805380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5889.807037] Lustre: Skipped 4 previous similar messages [ 5894.107753] Lustre: server umount lustre-MDT0000 complete [ 5894.869665] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5894.875989] LustreError: Skipped 6 previous similar messages [ 5895.014830] LustreError: 9904:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712766888 with bad export cookie 12827826358074975938 [ 5895.015747] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5895.021536] LustreError: 9904:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5895.119246] Lustre: server umount lustre-MDT0001 complete [ 5906.159877] Lustre: server umount lustre-OST0000 complete [ 5918.212829] Lustre: server umount lustre-OST0001 complete [ 5921.645134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5927.146382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5927.787242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5933.154993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5938.444846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5939.061312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5946.007526] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5949.425647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5949.434895] Lustre: lustre-MDT0000: reset Object Index mappings [ 5949.680996] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5949.714926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5951.137966] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5954.468359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5954.475801] Lustre: lustre-MDT0001: reset Object Index mappings [ 5955.922556] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5958.839185] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5958.844948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5960.704294] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5963.924032] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5963.931701] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5964.056999] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 5965.680634] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 5966.017854] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5973.689240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:318 to 0x280000401:354) [ 5973.694250] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 5979.959628] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5984.317641] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5985.922693] Lustre: MGS: Client 1e4781a1-b226-4c59-9fb0-c196b40b5407 (at 192.168.202.33@tcp) reconnecting [ 6011.315108] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 12:36:43 (1712767003) [ 6023.766309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.767251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6023.767252] Lustre: Skipped 3 previous similar messages [ 6023.774465] Lustre: Skipped 6 previous similar messages [ 6028.608791] Lustre: server umount lustre-MDT0000 complete [ 6028.774056] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6028.778193] LustreError: Skipped 3 previous similar messages [ 6029.764404] LustreError: 18003:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712767023 with bad export cookie 12827826358075022208 [ 6029.766504] LustreError: 166-1: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6029.779728] LustreError: 18003:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6035.900544] Lustre: server umount lustre-MDT0001 complete [ 6043.008928] Lustre: server umount lustre-OST0000 complete [ 6050.371683] Lustre: server umount lustre-OST0001 complete [ 6052.256720] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 6054.383814] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6057.305144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6059.815914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6062.095233] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6062.104282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6064.419587] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6064.427108] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6066.778381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6066.808608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6066.897143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6066.909382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6066.948586] Lustre: lustre-MDT0000: new disk, initializing [ 6066.997496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6068.085072] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6071.349933] Lustre: server umount lustre-MDT0000 complete [ 6072.462181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6074.756581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6074.785864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6074.923816] Lustre: Found index 0 for lustre-MDT0000, updating log [ 6074.930916] Lustre: Modifying parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6076.213557] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6078.080685] Lustre: server umount lustre-MDT0000 complete [ 6083.879496] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 12:37:55 (1712767075) [ 6084.731548] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 6091.629799] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 12:38:03 (1712767083) [ 6135.590507] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:38:47 (1712767127) [ 6141.997800] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 6144.846460] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6149.419865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6152.924481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6156.170698] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6156.181100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6160.089289] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6160.106041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6163.868823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6163.911014] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6164.078156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6164.082705] Lustre: Skipped 1 previous similar message [ 6164.100074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6164.102369] Lustre: Skipped 1 previous similar message [ 6164.165219] Lustre: lustre-MDT0000: new disk, initializing [ 6164.224441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6164.228572] Lustre: Skipped 5 previous similar messages [ 6164.241806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6166.179354] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6172.115049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6172.156065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6172.237737] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6172.240613] Lustre: Skipped 1 previous similar message [ 6172.297513] Lustre: lustre-MDT0001: new disk, initializing [ 6172.388777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6172.396911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6173.901402] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6177.364301] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6178.682533] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6182.013403] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6182.022765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6182.046180] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6182.052427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6182.223930] Lustre: lustre-OST0000: new disk, initializing [ 6182.239157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6183.877286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6183.880870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6183.925494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6184.721642] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6187.998440] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6192.004594] Lustre: lustre-MDD0000: changelog on [ 6193.338796] Lustre: lustre-MDD0001: changelog on [ 6199.994822] Lustre: lustre-MDT0000: Client 7f757d34-36bc-4221-ae97-e922e1bcc83b (at 192.168.202.33@tcp) reconnecting [ 6210.845810] Lustre: DEBUG MARKER: conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? [ 6210.989933] Lustre: 5306:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a5fb4000 x1795966022755456/t4294970631(0) o35->7f757d34-36bc-4221-ae97-e922e1bcc83b@192.168.202.33@tcp:640/0 lens 392/456 e 0 to 0 dl 1712767215 ref 1 fl Interpret:/202/0 rc 0/0 job:'createmany.0' uid:0 gid:0