[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 585368388 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] tsc: Detected 2400.000 MHz processor [ 1.055054] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 1.057979] pid_max: default: 32768 minimum: 301 [ 1.059700] Security Framework initialized [ 1.061616] SELinux: Initializing. [ 1.064694] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 1.076445] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 1.091309] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 1.117420] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 1.150634] Initializing cgroup subsys memory [ 1.173043] Initializing cgroup subsys devices [ 1.185857] Initializing cgroup subsys freezer [ 1.205260] Initializing cgroup subsys net_cls [ 1.215075] Initializing cgroup subsys blkio [ 1.227923] Initializing cgroup subsys perf_event [ 1.256037] Initializing cgroup subsys hugetlb [ 1.257439] Initializing cgroup subsys pids [ 1.261291] Initializing cgroup subsys net_prio [ 1.263140] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 1.368985] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 1.371272] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 1.373364] tlb_flushall_shift: 6 [ 1.374734] FEATURE SPEC_CTRL Present [ 1.376011] FEATURE IBPB_SUPPORT Present [ 1.377439] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 1.379357] Spectre V2 : Vulnerable [ 1.380654] Speculative Store Bypass: Vulnerable [ 1.383466] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 1.454839] ACPI: Core revision 20130517 [ 1.525929] ACPI: All ACPI Tables successfully acquired [ 1.531704] ftrace: allocating 30294 entries in 119 pages [ 1.731907] Enabling x2apic [ 1.732993] Enabled x2apic [ 1.740306] Switched APIC routing to physical x2apic. [ 1.825450] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 1.875900] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 1.887752] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 1.903318] ... version: 2 [ 1.906590] ... bit width: 48 [ 1.920563] ... generic registers: 4 [ 1.921654] ... value mask: 0000ffffffffffff [ 1.922955] ... max period: 00007fffffffffff [ 1.926539] ... fixed-purpose events: 3 [ 1.928088] ... event mask: 000000070000000f [ 1.930122] KVM setup paravirtual spinlock [ 1.934870] smpboot: Booting Node 0, Processors #1[ 1.936891] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 1.993661] KVM setup async PF for cpu 1 [ 2.006925] kvm-stealtime: cpu 1, msr 13e2935c0 #2[ 2.022920] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 2.052896] KVM setup async PF for cpu 2 #3 OK [ 2.062154] kvm-stealtime: cpu 2, msr 13e3135c0 [ 2.097658] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 2.164583] Brought up 4 CPUs [ 2.165514] KVM setup async PF for cpu 3 [ 2.165522] kvm-stealtime: cpu 3, msr 13e3935c0 [ 2.168722] smpboot: Max logical packages: 1 [ 2.170186] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 2.208356] devtmpfs: initialized [ 2.211332] x86/mm: Memory block size: 128MB [ 2.217410] EVM: security.selinux [ 2.219295] EVM: security.ima [ 2.220592] EVM: security.capability [ 2.275060] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 2.295875] NET: Registered protocol family 16 [ 2.314754] cpuidle: using governor haltpoll [ 2.335308] ACPI: bus type PCI registered [ 2.336702] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 2.349197] PCI: Using configuration type 1 for base access [ 2.360725] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 2.475368] ACPI: Added _OSI(Module Device) [ 2.480050] ACPI: Added _OSI(Processor Device) [ 2.481627] ACPI: Added _OSI(3.0 _SCP Extensions) [ 2.483450] ACPI: Added _OSI(Processor Aggregator Device) [ 2.486503] ACPI: Added _OSI(Linux-Dell-Video) [ 2.523311] ACPI: Interpreter enabled [ 2.530634] ACPI: (supports S0 S3 S4 S5) [ 2.532834] ACPI: Using IOAPIC for interrupt routing [ 2.535409] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 2.583759] ACPI: Enabled 2 GPEs in block 00 to 0F [ 2.633410] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 2.642690] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 2.667698] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 2.677264] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 2.683283] acpiphp: Slot [2] registered [ 2.695017] acpiphp: Slot [3] registered [ 2.696539] acpiphp: Slot [4] registered [ 2.698544] acpiphp: Slot [5] registered [ 2.721889] acpiphp: Slot [6] registered [ 2.724000] acpiphp: Slot [7] registered [ 2.726485] acpiphp: Slot [8] registered [ 2.728536] acpiphp: Slot [9] registered [ 2.730604] acpiphp: Slot [10] registered [ 2.732108] acpiphp: Slot [11] registered [ 2.733409] acpiphp: Slot [12] registered [ 2.767566] acpiphp: Slot [13] registered [ 2.792899] acpiphp: Slot [14] registered [ 2.841272] acpiphp: Slot [15] registered [ 2.842810] acpiphp: Slot [16] registered [ 2.844389] acpiphp: Slot [17] registered [ 2.849572] acpiphp: Slot [18] registered [ 2.852014] acpiphp: Slot [19] registered [ 2.853250] acpiphp: Slot [20] registered [ 2.855014] acpiphp: Slot [21] registered [ 2.856579] acpiphp: Slot [22] registered [ 2.859332] acpiphp: Slot [23] registered [ 2.860719] acpiphp: Slot [24] registered [ 2.863020] acpiphp: Slot [25] registered [ 2.864361] acpiphp: Slot [26] registered [ 2.865594] acpiphp: Slot [27] registered [ 2.867108] acpiphp: Slot [28] registered [ 2.868774] acpiphp: Slot [29] registered [ 2.870123] acpiphp: Slot [30] registered [ 2.876826] acpiphp: Slot [31] registered [ 2.878405] PCI host bridge to bus 0000:00 [ 2.882451] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 2.884613] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 2.890774] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 2.893292] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 2.896307] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 2.898681] pci_bus 0000:00: root bus resource [bus 00-ff] [ 2.940423] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 2.953334] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 2.970931] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 2.985600] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 3.005133] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 3.008300] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 3.456298] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 3.459208] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 3.461021] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 3.463160] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 3.464755] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 3.467944] vgaarb: loaded [ 3.468885] SCSI subsystem initialized [ 3.469941] ACPI: bus type USB registered [ 3.470824] usbcore: registered new interface driver usbfs [ 3.471973] usbcore: registered new interface driver hub [ 3.481006] usbcore: registered new device driver usb [ 3.490131] PCI: Using ACPI for IRQ routing [ 3.491980] NetLabel: Initializing [ 3.492868] NetLabel: domain hash size = 128 [ 3.494203] NetLabel: protocols = UNLABELED CIPSOv4 [ 3.496171] NetLabel: unlabeled traffic allowed by default [ 3.502671] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 3.505105] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 3.512769] amd_nb: Cannot enumerate AMD northbridges [ 3.525326] Switched to clocksource kvm-clock [ 3.572411] pnp: PnP ACPI init [ 3.573448] ACPI: bus type PNP registered [ 3.576645] pnp: PnP ACPI: found 6 devices [ 3.577943] ACPI: bus type PNP unregistered [ 3.595648] NET: Registered protocol family 2 [ 3.598402] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 3.602423] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 3.605263] TCP: Hash tables configured (established 32768 bind 32768) [ 3.608671] TCP: reno registered [ 3.609839] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 3.611879] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 3.614983] NET: Registered protocol family 1 [ 3.618414] RPC: Registered named UNIX socket transport module. [ 3.620239] RPC: Registered udp transport module. [ 3.621632] RPC: Registered tcp transport module. [ 3.622999] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 3.624970] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 3.626708] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 3.629111] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 3.634516] Unpacking initramfs... [ 6.865648] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 6.874771] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 6.877849] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 6.881995] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 6.884905] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 6.886481] RAPL PMU: hw unit of domain package 2^-0 Joules [ 6.888008] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 6.895264] cryptomgr_test (52) used greatest stack depth: 14480 bytes left [ 6.902706] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 6.904645] Initialise system trusted keyring [ 6.964549] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 6.967248] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 6.981697] zpool: loaded [ 6.983226] zbud: loaded [ 6.984529] VFS: Disk quotas dquot_6.6.0 [ 6.990435] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 7.018979] NFS: Registering the id_resolver key type [ 7.020391] Key type id_resolver registered [ 7.027712] Key type id_legacy registered [ 7.029207] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 7.039081] Key type big_key registered [ 7.059746] cryptomgr_test (58) used greatest stack depth: 14048 bytes left [ 7.072498] cryptomgr_test (63) used greatest stack depth: 13984 bytes left [ 7.078135] NET: Registered protocol family 38 [ 7.078147] Key type asymmetric registered [ 7.078150] Asymmetric key parser 'x509' registered [ 7.078272] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 7.078343] io scheduler noop registered [ 7.078347] io scheduler deadline registered (default) [ 7.078407] io scheduler cfq registered [ 7.078411] io scheduler mq-deadline registered [ 7.078416] io scheduler kyber registered [ 7.079922] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 7.079931] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 7.134680] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.142986] ACPI: Power Button [PWRF] [ 7.144396] GHES: HEST is not enabled! [ 7.263560] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 7.504938] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 7.783738] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 7.891900] tsc: Refined TSC clocksource calibration: 2399.948 MHz [ 8.034142] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 8.486854] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.564091] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 8.655567] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 8.672645] Non-volatile memory driver v1.3 [ 8.678115] Linux agpgart interface v0.103 [ 8.680951] crash memory driver: version 1.1 [ 8.683062] nbd: registered device at major 43 [ 8.762918] virtio_blk virtio1: [vda] 113176 512-byte logical blocks (57.9 MB/55.2 MiB) [ 8.807428] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.831920] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.849721] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.864385] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.911923] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.944147] rdac: device handler registered [ 8.964212] hp_sw: device handler registered [ 8.974167] emc: device handler registered [ 8.976410] libphy: Fixed MDIO Bus: probed [ 8.998570] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 9.005376] ehci-pci: EHCI PCI platform driver [ 9.010643] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 9.013138] ohci-pci: OHCI PCI platform driver [ 9.017745] uhci_hcd: USB Universal Host Controller Interface driver [ 9.019815] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 9.036355] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.038088] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 9.047742] mousedev: PS/2 mouse device common for all mice [ 9.056008] rtc_cmos 00:05: RTC can wake from S4 [ 9.061467] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 9.063947] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 9.067571] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 9.075174] hidraw: raw HID events driver (C) Jiri Kosina [ 9.080536] usbcore: registered new interface driver usbhid [ 9.085765] usbhid: USB HID core driver [ 9.088844] drop_monitor: Initializing network drop monitor service [ 9.091610] Netfilter messages via NETLINK v0.30. [ 9.097140] TCP: cubic registered [ 9.099500] Initializing XFRM netlink socket [ 9.110062] NET: Registered protocol family 10 [ 9.114726] NET: Registered protocol family 17 [ 9.118218] Key type dns_resolver registered [ 9.122300] mce: Using 10 MCE banks [ 9.125281] Loading compiled-in X.509 certificates [ 9.130114] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 9.135561] registered taskstats version 1 [ 9.149114] modprobe (72) used greatest stack depth: 13456 bytes left [ 9.158607] Key type trusted registered [ 9.177750] Key type encrypted registered [ 9.179911] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 9.188945] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 9.197565] rtc_cmos 00:05: setting system clock to 2024-12-15 09:29:15 UTC (1734254955) [ 9.204592] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 9.210116] Write protecting the kernel read-only data: 12288k [ 9.214133] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 9.218692] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 9.230592] random: systemd: uninitialized urandom read (16 bytes read) [ 9.241960] random: systemd: uninitialized urandom read (16 bytes read) [ 9.245036] random: systemd: uninitialized urandom read (16 bytes read) [ 9.249772] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 9.258166] systemd[1]: Detected virtualization kvm. [ 9.260663] systemd[1]: Detected architecture x86-64. [ 9.262996] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-572.el7 (Initramfs)! [ 9.272504] systemd[1]: No hostname configured. [ 9.273986] systemd[1]: Set hostname to . [ 9.276029] random: systemd: uninitialized urandom read (16 bytes read) [ 9.278243] systemd[1]: Initializing machine ID from random generator. [ 9.415677] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 9.420022] random: systemd: uninitialized urandom read (16 bytes read) [ 9.423537] random: systemd: uninitialized urandom read (16 bytes read) [ 9.432234] random: systemd: uninitialized urandom read (16 bytes read) [ 9.438639] random: systemd: uninitialized urandom read (16 bytes read) [ 9.447328] random: systemd: uninitialized urandom read (16 bytes read) [ 9.456684] random: systemd: uninitialized urandom read (16 bytes read) [ 9.498003] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 9.516787] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 9.532473] systemd[1]: Created slice Root Slice. [ OK ] Created slice Root Slice. [ 9.544373] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 9.560686] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 9.569690] systemd[1]: Created slice System Slice. [ OK ] Created slice System Slice. [ 9.583391] systemd[1]: Starting Journal Service... Starting Journal Service... [ 9.602939] systemd[1]: Starting Load Kernel Modules... Starting Load Kernel Modules... [ 9.637387] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ 9.664984] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 9.682265] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 9.692248] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 9.710299] systemd[1]: Reached target Sockets. [ OK ] Reached target Sockets. [ 9.724078] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 9.773803] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 9.793634] systemd[1]: Started Journal Service. [ OK ] Started Journal Service. [ OK ] Started Load Kernel Modules. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting Create Static Device Nodes in /dev... Starting Apply Kernel Variables... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Apply Kernel Variables. [ 10.021907] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Mounting Configuration File System... [ 11.585768] random: fast init done Starting Show Plymouth Boot Screen... Starting dracut initqueue hook... [ OK ] Mounted Configuration File System. [ OK ] Reached target System Initialization. [ OK ] Started Show Plymouth Boot Screen. [ OK ] Started Forward Password Requests to Plymouth Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. %G[ 12.186594] scsi host0: ata_piix [ 12.203045] scsi host1: ata_piix [ 12.209741] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.215181] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.607573] ip (323) used greatest stack depth: 13080 bytes left [ 12.817179] ip (346) used greatest stack depth: 12336 bytes left [ 12.593529] dracut-initqueue[280]: RTNETLINK answers: File exists [ 13.712074] dracut-initqueue[280]: bs=4096, sz=32212254720 bytes [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... Mounting /sysroot... [ 17.118663] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. [ OK ] Mounted /sysroot. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Load Kernel Modules. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Plymouth switch root service. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.038891] systemd-journald[100]: Received SIGTERM from PID 1 (systemd). [ 18.502613] SELinux: Disabled at runtime. [ 18.635670] ip_tables: (C) 2000-2006 Netfilter Core Team [ 18.642974] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Stopped Journal Service. Starting Journal Service... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st... nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. Mounting Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Starting Load Kernel Modules... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. Starting Set Up Additional Binary Formats... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on /dev/initctl Compatibility Named Pipe. Starting Read and set NIS domainname from /etc/sysconfig/network... [ OK ] Stopped target Initrd File Systems. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Debug File System. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Load Kernel Modules. Mounting Arbitrary Executable File Formats File System... Starting Apply Kernel Variables... Starting Create Static Device Nodes in /dev... [ OK ] Started Read and set NIS domainname from /etc/sysconfig/network. [ OK ] Started Journal Service. [ OK ] Mounted Arbitrary Executable File Formats File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started udev Coldplug all Devices. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Set Up Additional Binary Formats. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.253701] systemd-journald[570]: Received request to flush runtime journal from PID 1 [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Kernel Device Manager. [ 20.968697] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ OK ] Found device /dev/ttyS1. [ OK ] Found device /dev/ttyS0. [ 21.195358] cryptd: max_cpu_qlen set to 1000 [ 21.249190] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ OK ] Found device /dev/vda. Mounting /home/green/git/lustre-release... [ OK ] Found device /dev/disk/by-label/SWAP. Activating swap /dev/disk/by-label/SWAP... [ 21.850669] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS %G[ OK ] Activated swap /dev/disk/by-label/SWAP. [ 21.882766] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Reached target Swap. [ 21.911682] AVX version of gcm_enc/dec engaged. [ 21.913124] AES CTR mode by8 optimization enabled [ OK ] Mounted /home/green/git/lustre-release. [ 21.952452] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 21.968386] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 22.397206] EDAC MC: Ver: 3.0.0 [ 22.433351] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur...nly root support (7s / no limit) [** ] A start job is running for Configur...nly root support (8s / no limit) [*** ] A start job is running for Configur...nly root support (8s / no limit) [ *** ] A start job is running for Configur...nly root support (9s / no limit) [ *** ] A start job is running for Configur...nly root support (9s / no limit) [ ***] A start job is running for Configur...ly root support (10s / no limit) [ **] A start job is running for Configur...ly root support (10s / no limit) [ *] A start job is running for Configur...ly root support (11s / no limit) [ **] A start job is running for Configur...ly root support (11s / no limit) [ ***] A start job is running for Configur...ly root support (12s / no limit) [ *** ] A start job is running for Configur...ly root support (12s / no limit) [ *** ] A start job is running for Configur...ly root support (13s / no limit)[ 32.816732] mount.nfs (783) used greatest stack depth: 10704 bytes left [*** ] A start job is running for Configur...ly root support (13s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Preprocess NFS configuration... Starting Tell Plymouth To Write Out Runtime Data... Starting Rebuild Journal Catalog... [ 33.097244] hrtimer: interrupt took 4416160 ns Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Preprocess NFS configuration. [ OK ] Started Tell Plymouth To Write Out Runtime Data. [FAILED] Failed to start Create Volatile Files and Directories. See 'systemctl status systemd-tmpfiles-setup.service' for details. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... [FAILED] Failed to start Rebuild Journal Catalog. See 'systemctl status systemd-journal-catalog-update.service' for details. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Started Flexible branding. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting GSSAPI Proxy Daemon... Starting Dump dmesg to /var/log/dmesg... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Dump dmesg to /var/log/dmesg. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting /etc/rc.d/rc.local Compatibility... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started /etc/rc.d/rc.local Compatibility. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... Starting Terminate Plymouth Boot Screen... Starting Wait for Plymouth Boot Screen to Quit... CentOS Linux 7 (Core) Kernel 3.10.0-7.9-debug on an x86_64 oleg430-server login: [ 54.670705] device-mapper: uevent: version 1.0.3 [ 54.677709] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 63.879432] libcfs: loading out-of-tree module taints kernel. [ 63.881364] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 63.947257] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 72.006082] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 72.433811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 72.440173] alg: No test for adler32 (adler32-zlib) [ 73.471496] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 73.752107] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 73.756571] LNet: Accept secure, port 988 [ 75.373920] Key type lgssc registered [ 75.986726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 82.076067] icp: module license 'CDDL' taints kernel. [ 82.077473] Disabling lock debugging due to kernel taint [ 84.865994] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 90.819706] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.326712] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.284001] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 106.294453] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.750958] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 111.772173] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 118.580678] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 125.416374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.445648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 125.464593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 126.655227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 126.679329] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 126.750017] Lustre: lustre-MDT0000: new disk, initializing [ 126.790891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 126.802267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 126.862236] mount.lustre (7244) used greatest stack depth: 10224 bytes left [ 128.462183] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 136.150429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.197334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 136.250343] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 136.261241] Lustre: 8404:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 136.271553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 136.273983] Lustre: Skipped 1 previous similar message [ 136.321965] Lustre: lustre-MDT0001: new disk, initializing [ 136.360359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 136.388532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 136.395179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 138.126354] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 141.372826] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 146.129479] random: crng init done [ 146.255837] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 146.262782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 146.314154] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 146.321992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 146.441114] Lustre: 9608:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 146.466085] Lustre: lustre-OST0000: new disk, initializing [ 146.476750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 146.527434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 149.028301] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 150.802149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 150.820003] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 150.849283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 157.111101] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 157.119907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 157.158433] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 157.164740] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 157.211726] Lustre: 10884:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 157.227780] Lustre: lustre-OST0001: new disk, initializing [ 157.230980] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 157.269761] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.622679] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.205344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 160.208655] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 160.228279] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 166.823517] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 172.197035] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 178.858048] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing check_logdir /tmp/testlogs/ [ 179.990939] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing yml_node [ 181.323318] Lustre: DEBUG MARKER: Client: 2.16.50.90 [ 182.129770] Lustre: DEBUG MARKER: MDS: 2.16.50.90 [ 182.929851] Lustre: DEBUG MARKER: OSS: 2.16.50.90 [ 183.377409] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity-1 ============----- Sun Dec 15 04:32:08 EST 2024 [ 189.769976] Lustre: DEBUG MARKER: excepting tests: [ 190.524297] Lustre: DEBUG MARKER: skipping tests SLOW=no: 106 [ 205.276734] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 205.277376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 205.285694] Lustre: Skipped 3 previous similar messages [ 207.730132] Lustre: server umount lustre-MDT0000 complete [ 209.122290] LustreError: 7252:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255155 with bad export cookie 6227988356773479337 [ 209.122556] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 209.131378] LustreError: 7252:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 210.284472] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 210.285758] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 210.285759] Lustre: Skipped 3 previous similar messages [ 210.300918] Lustre: Skipped 1 previous similar message [ 214.292941] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 214.296575] Lustre: Skipped 2 previous similar messages [ 215.308888] Lustre: server umount lustre-MDT0001 complete [ 216.950826] Lustre: server umount lustre-OST0000 complete [ 218.661869] Lustre: server umount lustre-OST0001 complete [ 220.864759] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 223.380031] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 227.063713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 229.956750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 232.863641] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 232.875052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 235.952028] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 235.962788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 239.438271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 239.490443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 239.670204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 239.695737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 239.753385] Lustre: lustre-MDT0000: new disk, initializing [ 239.812705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 239.832331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 239.902678] mount.lustre (17665) used greatest stack depth: 10128 bytes left [ 241.527393] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 247.698336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 247.749494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 247.821716] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 247.848175] Lustre: 18623:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 247.869373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 247.872033] Lustre: Skipped 1 previous similar message [ 247.917423] Lustre: lustre-MDT0001: new disk, initializing [ 247.982730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 248.006914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 248.013847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 248.039787] mount.lustre (18623) used greatest stack depth: 9520 bytes left [ 249.410768] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 253.091105] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 254.742657] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 257.971120] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 257.977185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 258.013602] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 258.020116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 258.116709] Lustre: 19821:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 259.745037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 259.804574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 259.849551] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 263.577357] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 265.322003] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 265.458490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 267.138243] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 267.258463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 269.788867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 269.791234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.791626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 274.325619] Lustre: server umount lustre-OST0000 complete [ 278.030024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 278.033385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 278.038532] Lustre: Skipped 1 previous similar message [ 278.041115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 278.043736] Lustre: Skipped 1 previous similar message [ 282.508741] Lustre: server umount lustre-MDT0000 complete [ 283.788046] LustreError: 17675:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255230 with bad export cookie 6227988356773480870 [ 283.789447] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 283.797345] LustreError: 17675:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 283.921364] Lustre: server umount lustre-MDT0001 complete [ 285.153435] Lustre: DEBUG MARKER: == conf-sanity-1 test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ========================================================== 04:33:50 (1734255230) [ 287.630565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 287.823858] LustreError: 22449:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.832669] LustreError: 22449:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 5 previous similar messages [ 287.866469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 287.869096] Lustre: Skipped 1 previous similar message [ 288.831845] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 291.785958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 291.849174] Lustre: 23309:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 293.229274] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 295.172175] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 296.596553] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 299.578456] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 299.593239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 301.656394] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 303.592685] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 316.780434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 316.782759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.786724] Lustre: Skipped 1 previous similar message [ 316.789102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 316.791127] Lustre: Skipped 2 previous similar messages [ 321.843450] Lustre: server umount lustre-OST0000 complete [ 326.972503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 326.975379] LustreError: Skipped 1 previous similar message [ 326.977062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.981388] Lustre: Skipped 1 previous similar message [ 328.868964] Lustre: server umount lustre-MDT0000 complete [ 329.926587] LustreError: 22431:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255276 with bad export cookie 6227988356773481752 [ 329.927700] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 329.936300] LustreError: 22431:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 330.037308] Lustre: server umount lustre-MDT0001 complete [ 333.232295] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 333.855041] Key type lgssc unregistered [ 333.951247] LNet: 26302:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.957173] LNet: Removed LNI 192.168.204.130@tcp [ 340.995178] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 341.162956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 341.208118] alg: No test for adler32 (adler32-zlib) [ 342.037789] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 342.097759] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 342.100670] LNet: Accept secure, port 988 [ 343.641918] Key type lgssc registered [ 343.964614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 346.606698] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 346.615798] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 346.622873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 363.725055] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 369.727963] LustreError: 27597:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.130@tcp: failed processing log, type 4: rc = -110 [ 399.743221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.230219] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 402.879333] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 405.257964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.425485] LustreError: MGC192.168.204.130@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 405.432196] LustreError: 28766:0:(tgt_mount.c:1493:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 405.436757] LustreError: 28766:0:(tgt_mount.c:2222:server_fill_super()) Unable to start targets: -2 [ 405.440613] LustreError: 28766:0:(tgt_mount.c:1721:server_put_super()) no obd lustre-MDT0000 [ 405.505685] Lustre: server umount lustre-MDT0000 complete [ 405.507923] LustreError: 28766:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 418.749643] Lustre: server umount lustre-OST0000 complete [ 422.240507] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 424.472578] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 427.127265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 428.975470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 430.872622] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 430.878489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 432.735817] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 432.746804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 434.897228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 434.920211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.013831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 435.026815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 435.064569] Lustre: lustre-MDT0000: new disk, initializing [ 435.094152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.102083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 436.020560] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 439.703301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 439.728826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 439.764509] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 439.775545] Lustre: 670:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 439.790176] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 439.792025] Lustre: Skipped 1 previous similar message [ 439.824116] Lustre: lustre-MDT0001: new disk, initializing [ 439.853027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 439.864510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 439.868510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 440.742374] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 443.326281] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 444.158357] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 445.982816] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 445.986463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 446.004330] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 446.007523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 446.071587] Lustre: 1894:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 446.084016] Lustre: lustre-OST0000: new disk, initializing [ 446.085686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 446.105226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.447005] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 449.967494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.971003] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 449.999862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 450.136684] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 450.952401] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 450.998782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 451.849343] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 451.903663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 455.004735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.005062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 455.006657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 455.014372] Lustre: Skipped 1 previous similar message [ 458.451031] Lustre: server umount lustre-OST0000 complete [ 459.868634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 459.872095] LustreError: Skipped 1 previous similar message [ 459.874159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.879850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 459.882078] Lustre: Skipped 1 previous similar message [ 465.020353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 465.022869] Lustre: Skipped 3 previous similar messages [ 465.505867] Lustre: server umount lustre-MDT0000 complete [ 466.478682] LustreError: 683:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255412 with bad export cookie 9568861182452041001 [ 466.480025] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 466.487981] LustreError: 683:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 466.584406] Lustre: server umount lustre-MDT0001 complete [ 468.653088] Lustre: DEBUG MARKER: == conf-sanity-1 test 18: check mkfs creates large journals ========================================================== 04:36:54 (1734255414) [ 469.415641] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 473.306602] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 474.914639] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 477.778089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 480.137447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 481.817461] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 481.822098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 483.848446] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 483.855450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 486.039666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 486.076723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.172877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 486.182881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 486.216887] Lustre: lustre-MDT0000: new disk, initializing [ 486.248847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 486.258121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 487.226042] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 491.543876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 491.570389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 491.601768] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 491.616082] Lustre: 8696:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 491.627372] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 491.628989] Lustre: Skipped 1 previous similar message [ 491.674035] Lustre: lustre-MDT0001: new disk, initializing [ 491.675483] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 491.729302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 491.732889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 492.796959] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 495.589583] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 496.458905] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 498.275964] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 498.280190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 498.301012] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 498.304641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 498.371045] Lustre: 9893:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 498.398850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.400963] Lustre: Skipped 1 previous similar message [ 499.727829] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 502.254664] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 503.079302] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 506.869944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 506.872298] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 506.884691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 507.174217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 507.911462] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 507.965586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 511.884428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.884641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 511.884901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 511.893846] Lustre: Skipped 2 previous similar messages [ 514.495938] Lustre: server umount lustre-OST0000 complete [ 516.748501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 516.751173] LustreError: Skipped 1 previous similar message [ 516.752651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.568591] Lustre: server umount lustre-MDT0000 complete [ 522.614168] LustreError: 7747:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255468 with bad export cookie 9568861182452041869 [ 522.615523] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.621924] LustreError: 7747:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 522.720609] Lustre: server umount lustre-MDT0001 complete [ 524.947127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.118107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.926483] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 528.091275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 528.121851] Lustre: 13346:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 528.954225] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 530.231877] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 531.061229] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 532.762818] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 532.765726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 534.060135] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 535.375851] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 542.115387] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 542.860618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.861016] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 542.869171] Lustre: Skipped 1 previous similar message [ 542.871590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 542.873139] Lustre: Skipped 4 previous similar messages [ 544.819949] Lustre: server umount lustre-OST0000 complete [ 547.915221] Lustre: server umount lustre-MDT0000 complete [ 548.818527] LustreError: 15648:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255495 with bad export cookie 9568861182452042751 [ 548.819805] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.827762] LustreError: 15648:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 548.916667] Lustre: server umount lustre-MDT0001 complete [ 551.899707] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 552.477117] Key type lgssc unregistered [ 552.568155] LNet: 16448:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 553.574394] LNet: Removed LNI 192.168.204.130@tcp [ 558.426789] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 562.757005] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 562.921854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 562.927928] alg: No test for adler32 (adler32-zlib) [ 563.745132] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 563.792966] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 563.794847] LNet: Accept secure, port 988 [ 565.321132] Key type lgssc registered [ 565.538488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 567.766339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 569.543825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 571.229143] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 571.237176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 572.870895] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 572.876644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 574.769425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 574.785423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 574.790585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.871430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 575.890510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 575.926462] Lustre: lustre-MDT0000: new disk, initializing [ 575.953521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.962259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 576.931406] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 580.654381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 580.677924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 580.704868] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 580.714873] Lustre: 21110:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 580.728033] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 580.730075] Lustre: Skipped 1 previous similar message [ 580.763877] Lustre: lustre-MDT0001: new disk, initializing [ 580.794502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 580.809234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 580.814121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 581.689887] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 584.329480] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 585.245950] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 587.063514] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 587.070053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 587.088171] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 587.091356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 587.174071] Lustre: 22304:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 587.188279] Lustre: lustre-OST0000: new disk, initializing [ 587.190385] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 587.225292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 588.588701] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 591.108997] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 591.901417] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 594.001405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 594.005887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 594.028468] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 595.005570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 595.873925] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 595.928402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 599.020416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 599.020885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.021192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 602.477329] Lustre: server umount lustre-OST0000 complete [ 604.028476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.028740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 604.028741] Lustre: Skipped 1 previous similar message [ 604.036047] Lustre: Skipped 2 previous similar messages [ 609.036278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 609.039219] Lustre: Skipped 2 previous similar messages [ 609.499744] Lustre: server umount lustre-MDT0000 complete [ 610.533974] LustreError: 20164:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255556 with bad export cookie 10988198205420257362 [ 610.534763] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 610.541734] LustreError: 20164:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 610.653900] Lustre: server umount lustre-MDT0001 complete [ 612.679507] Lustre: DEBUG MARKER: == conf-sanity-1 test 21c: start mds between two osts, stop mds last ========================================================== 04:39:18 (1734255558) [ 614.283979] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 614.288350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 631.347157] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 637.349151] LustreError: 25121:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.130@tcp: failed processing log, type 4: rc = -110 [ 666.368028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 667.962775] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 669.754378] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 672.029501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.251835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 673.345920] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 676.207624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 676.242817] Lustre: 27171:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 677.290516] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 678.821353] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 679.862348] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 682.015517] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 682.019354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 682.042941] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 682.047137] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 682.076957] Lustre: 28285:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 682.088061] Lustre: lustre-OST0001: new disk, initializing [ 682.090193] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 682.114111] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 682.116972] Lustre: Skipped 1 previous similar message [ 683.730513] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 687.097691] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 688.498725] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 689.125983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 689.133217] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 689.168032] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 689.611599] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 690.713952] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 690.797325] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 692.252510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 692.257170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 692.268656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 697.536477] Lustre: server umount lustre-OST0000 complete [ 699.164820] LustreError: 25138:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 699.166364] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 699.166376] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 699.166377] Lustre: Skipped 1 previous similar message [ 699.166777] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 699.166778] Lustre: Skipped 2 previous similar messages [ 699.186193] LustreError: 25138:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 704.172979] LustreError: 25136:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 704.180661] LustreError: 25136:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 704.995759] Lustre: server umount lustre-OST0001 complete [ 706.348401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 706.351546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.355468] Lustre: Skipped 1 previous similar message [ 709.180371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.183908] Lustre: Skipped 5 previous similar messages [ 712.136565] Lustre: server umount lustre-MDT0000 complete [ 713.152650] LustreError: 25129:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255659 with bad export cookie 10988198205420258321 [ 713.153982] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 713.160188] LustreError: 25129:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 713.263768] Lustre: server umount lustre-MDT0001 complete [ 714.747072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 715.792055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 716.883204] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 716.886543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 717.907060] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 717.911182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 719.505828] Lustre: DEBUG MARKER: == conf-sanity-1 test 29: permanently remove an OST ====== 04:41:04 (1734255664) [ 721.411344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.527308] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 721.543179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 721.625764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 722.550099] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 724.895375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 724.923245] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 724.931815] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 724.938351] Lustre: 892:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 725.841391] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 727.329252] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 728.216866] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 730.205430] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 730.213524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 730.300843] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 731.795550] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 733.296129] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 741.497538] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 741.503130] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 741.550050] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 741.584543] Lustre: 3159:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 741.588945] Lustre: 3159:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 743.133347] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 744.736447] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 747.385554] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 757.807281] Lustre: Permanently deactivating lustre-OST0001 [ 757.815577] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 766.968319] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 771.413451] Lustre: Permanently reactivating lustre-OST0001 [ 771.436867] Lustre: Modifying parameter lustre-OST0001-osc.osc.active in log lustre-client [ 771.441017] Lustre: Skipped 2 previous similar messages [ 772.616061] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 772.620966] Lustre: Skipped 1 previous similar message [ 772.623619] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 772.627601] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 772.634170] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 772.644964] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 781.981522] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 784.684745] Lustre: server umount lustre-OST0001 complete [ 786.988982] LustreError: 2028:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 786.993557] LustreError: 2028:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 791.870531] Lustre: server umount lustre-OST0000 complete [ 795.100667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 795.104508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 795.108833] Lustre: Skipped 5 previous similar messages [ 799.208344] Lustre: server umount lustre-MDT0000 complete [ 800.434146] LustreError: 910:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255746 with bad export cookie 10988198205420259273 [ 800.436454] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 800.444622] LustreError: 910:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 800.595422] Lustre: server umount lustre-MDT0001 complete [ 805.307462] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 806.725547] Key type lgssc unregistered [ 806.883460] LNet: 5752:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.895274] LNet: Removed LNI 192.168.204.130@tcp [ 821.117881] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 827.152254] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 827.423108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.458245] alg: No test for adler32 (adler32-zlib) [ 828.311470] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 828.407812] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 828.409544] LNet: Accept secure, port 988 [ 829.970922] Key type lgssc registered [ 830.430774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 834.837790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 838.002440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 840.675393] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 840.684809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 843.679661] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 843.688273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 846.920452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 846.946988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 846.955494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 848.064648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 848.082493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 848.116080] Lustre: lustre-MDT0000: new disk, initializing [ 848.144846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.154887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 849.494676] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 854.302481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 854.342239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 854.377964] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 854.388604] Lustre: 11515:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 854.406276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 854.409544] Lustre: Skipped 1 previous similar message [ 854.453861] Lustre: lustre-MDT0001: new disk, initializing [ 854.509145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 854.528054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 854.540845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 855.820638] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 858.971128] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 860.258539] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 862.720164] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 862.727594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 862.753630] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 862.760304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 862.875524] Lustre: 12712:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 862.892538] Lustre: lustre-OST0000: new disk, initializing [ 862.895051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 862.928346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 864.810673] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 868.147418] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 869.432160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 869.440451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 869.473846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 869.629511] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 869.709700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 871.086994] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 871.158282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 874.462375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.464666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 874.471827] Lustre: Skipped 1 previous similar message [ 877.969029] Lustre: server umount lustre-OST0000 complete [ 879.470855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 879.472783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 879.472811] Lustre: Skipped 1 previous similar message [ 879.483137] Lustre: Skipped 1 previous similar message [ 884.477725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 884.479931] Lustre: Skipped 2 previous similar messages [ 885.370160] Lustre: server umount lustre-MDT0000 complete [ 886.879722] LustreError: 11530:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255833 with bad export cookie 6399050485767512188 [ 886.880980] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 886.890731] LustreError: 11530:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 887.061044] Lustre: server umount lustre-MDT0001 complete [ 890.284977] Lustre: DEBUG MARKER: == conf-sanity-1 test 35a: Reconnect to the last active server first ========================================================== 04:43:55 (1734255835) [ 893.387661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.635847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 895.011752] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 898.546213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 898.637281] LustreError: 15536:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 898.642343] Lustre: 16395:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 898.658620] LustreError: 15536:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 898.778695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 900.134731] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 902.308269] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 903.582710] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 906.648529] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 906.655750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 906.817927] Lustre: 17509:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 906.911232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 909.875644] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 913.250360] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 920.967121] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 922.569690] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 928.153018] LNetError: Refusing connection from 192.168.204.130 for 127.0.0.2@tcp: No matching NI [ 928.157554] LNetError: 8349:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 928.164317] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 933.201248] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2024-12-15 4h44m38s [ 933.865452] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 937.949712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 937.951872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 937.956886] Lustre: Skipped 2 previous similar messages [ 937.959237] LNetError: Refusing connection from 192.168.204.130 for 127.0.0.2@tcp: No matching NI [ 937.962404] LNetError: 8350:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 937.965749] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 940.767706] Lustre: server umount lustre-MDT0000 complete [ 942.101155] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 942.961371] LustreError: 16957:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 942.961657] LNetError: Refusing connection from 192.168.204.130 for 127.0.0.2@tcp: No matching NI [ 942.962222] LNetError: 8351:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 942.962241] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 942.981731] LustreError: 16957:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 944.460035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 944.538048] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 944.643663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 945.684429] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 946.831167] Lustre: DEBUG MARKER: Wait for df (9041) ... [ 948.644254] LNetError: Refusing connection from 192.168.204.130 for 127.0.0.2@tcp: No matching NI [ 948.647364] LNetError: 8352:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 948.647728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 948.658628] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 949.074638] Lustre: DEBUG MARKER: done [ 953.644943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.646575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.646577] Lustre: Skipped 3 previous similar messages [ 953.655241] Lustre: Skipped 1 previous similar message [ 956.023983] Lustre: server umount lustre-OST0000 complete [ 958.652703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.658259] Lustre: Skipped 1 previous similar message [ 958.660131] LNetError: Refusing connection from 192.168.204.130 for 127.0.0.2@tcp: No matching NI [ 958.663018] LNetError: 8349:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 958.665861] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 963.625314] Lustre: server umount lustre-MDT0000 complete [ 963.662723] LustreError: 16957:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 963.677684] LustreError: 16957:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 965.309155] LustreError: 15517:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255911 with bad export cookie 6399050485767513931 [ 965.309925] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 965.317510] LustreError: 15517:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 965.458045] Lustre: server umount lustre-MDT0001 complete [ 970.141978] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 971.531887] Key type lgssc unregistered [ 971.676565] LNet: 21124:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.688767] LNet: Removed LNI 192.168.204.130@tcp [ 987.232238] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 993.530172] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 993.722445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 993.729106] alg: No test for adler32 (adler32-zlib) [ 994.582588] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 994.646684] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 994.649154] LNet: Accept secure, port 988 [ 996.193631] Key type lgssc registered [ 996.601080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1000.010212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1002.432223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1004.559272] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1004.563737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1006.452547] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1006.459605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1008.553161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1008.570321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1008.576276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.668974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1009.687054] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1009.729667] Lustre: lustre-MDT0000: new disk, initializing [ 1009.774657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1009.785033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1010.931926] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1014.726732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1014.752885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1014.787400] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1014.795913] Lustre: 26882:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1014.809609] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1014.811865] Lustre: Skipped 1 previous similar message [ 1014.846561] Lustre: lustre-MDT0001: new disk, initializing [ 1014.880072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1014.895295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1014.900630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1015.816253] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1018.555712] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1019.423772] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1021.349491] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1021.354080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1021.371116] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1021.374408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1021.444629] Lustre: 28077:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1021.456777] Lustre: lustre-OST0000: new disk, initializing [ 1021.459143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1021.482859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1022.873854] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1025.680992] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1026.055776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1026.059974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1026.083814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1026.600689] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1026.658745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1027.658337] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1027.719230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1031.084777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.085367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1031.086806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1031.095462] Lustre: Skipped 1 previous similar message [ 1034.385592] Lustre: server umount lustre-OST0000 complete [ 1036.093517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.094627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1036.094629] Lustre: Skipped 1 previous similar message [ 1036.106038] Lustre: Skipped 1 previous similar message [ 1041.100825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1041.103368] Lustre: Skipped 2 previous similar messages [ 1041.846458] Lustre: server umount lustre-MDT0000 complete [ 1043.460457] LustreError: 25930:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255989 with bad export cookie 12608993017155219189 [ 1043.461937] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1043.469041] LustreError: 25930:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1043.633736] Lustre: server umount lustre-MDT0001 complete [ 1046.788450] Lustre: DEBUG MARKER: == conf-sanity-1 test 35b: Continue reconnection retries, if the active server is busy ========================================================== 04:46:32 (1734255992) [ 1049.173285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1049.415617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.469476] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1053.074424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1053.115365] Lustre: 31758:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1054.066997] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1055.562985] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1056.450541] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1058.276421] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1058.282093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1058.347681] Lustre: 411:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1058.396233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1058.399238] Lustre: Skipped 1 previous similar message [ 1058.416738] mount.lustre (411) used greatest stack depth: 9360 bytes left [ 1059.984359] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1061.956884] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1069.056806] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2024-12-15 4h46m54s [ 1069.677231] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1072.083592] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1073.020957] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1073.038990] LustreError: 30893:0:(mdt_handler.c:2354:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1075.161913] LNetError: Refusing connection from 192.168.204.130 for 127.0.0.2@tcp: No matching NI [ 1075.165457] LNetError: 23723:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1075.168990] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1093.068402] Lustre: lustre-MDT0000: Client 63ef587d-020f-413c-9151-bda905f52a61 (at 192.168.204.30@tcp) reconnecting [ 1113.100152] Lustre: lustre-MDT0000: Client 63ef587d-020f-413c-9151-bda905f52a61 (at 192.168.204.30@tcp) reconnecting [ 1113.105138] LustreError: 30893:0:(mdt_handler.c:2354:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1113.621348] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1114.454345] Lustre: DEBUG MARKER: done [ 1118.478192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.481120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1118.495373] Lustre: Skipped 1 previous similar message [ 1121.903478] Lustre: server umount lustre-OST0000 complete [ 1123.292474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1123.296347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.303410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1123.306768] Lustre: Skipped 1 previous similar message [ 1123.309205] LNetError: Refusing connection from 192.168.204.130 for 127.0.0.2@tcp: No matching NI [ 1123.312101] LNetError: 23726:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1123.315567] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1128.311679] LNetError: Refusing connection from 192.168.204.130 for 127.0.0.2@tcp: No matching NI [ 1128.316907] LNetError: 23724:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1128.323555] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1129.098409] Lustre: server umount lustre-MDT0000 complete [ 1129.314904] LustreError: 389:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.321132] LustreError: 389:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 1130.443760] LustreError: 434:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256076 with bad export cookie 12608993017155220057 [ 1130.444852] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.455571] LustreError: 434:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1130.566138] Lustre: server umount lustre-MDT0001 complete [ 1134.811929] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 1135.629224] Key type lgssc unregistered [ 1135.733545] LNet: 3330:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.744474] LNet: Removed LNI 192.168.204.130@tcp [ 1148.529250] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 1154.239118] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 1154.534521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1154.558718] alg: No test for adler32 (adler32-zlib) [ 1155.431858] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1155.491842] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 1155.495359] LNet: Accept secure, port 988 [ 1157.047104] Key type lgssc registered [ 1157.407549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1162.001272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1164.578821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1166.980017] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1166.989253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1169.598493] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1169.605154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1171.834346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1171.854674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1171.862857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1172.945702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1172.958484] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1172.998443] Lustre: lustre-MDT0000: new disk, initializing [ 1173.037989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.046604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1174.014405] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1178.204687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1178.236203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1178.271529] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1178.282662] Lustre: 9092:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1178.298305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1178.301119] Lustre: Skipped 1 previous similar message [ 1178.335967] Lustre: lustre-MDT0001: new disk, initializing [ 1178.377420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1178.392434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1178.398474] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1179.437061] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1182.293259] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1183.311155] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1185.362654] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1185.368546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1185.399254] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1185.403811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1185.479632] Lustre: 10288:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1185.491928] Lustre: lustre-OST0000: new disk, initializing [ 1185.493923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1185.517681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1186.910586] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1187.303136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1187.307137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1187.332400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1189.547198] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1190.360923] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1190.410589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1191.253244] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1191.309902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1192.332973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1192.333027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1192.333484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1192.341813] LustreError: Skipped 1 previous similar message [ 1197.340700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1197.342958] Lustre: Skipped 2 previous similar messages [ 1197.898691] Lustre: server umount lustre-OST0000 complete [ 1202.348414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.348735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1202.354138] Lustre: Skipped 2 previous similar messages [ 1205.055299] Lustre: server umount lustre-MDT0000 complete [ 1206.104787] LustreError: 8144:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256152 with bad export cookie 15444839865238167937 [ 1206.106031] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1206.112641] LustreError: 8144:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1206.219902] Lustre: server umount lustre-MDT0001 complete [ 1208.473471] Lustre: DEBUG MARKER: == conf-sanity-1 test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:49:13 (1734256153) [ 1216.677479] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 1218.686205] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 1221.462126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1223.381156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1225.370488] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1225.376016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1227.281608] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1227.289692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1229.447214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1229.475779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.621289] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1234.835231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1234.869750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1234.972568] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1234.979860] Lustre: 17649:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1234.997268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1235.041560] Lustre: lustre-MDT0001: new disk, initializing [ 1235.085068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1235.098725] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1235.101307] Lustre: 17687:0:(lod_sub_object.c:948:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1235.105252] LustreError: 17687:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1236.192684] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1240.301515] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1240.306245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1240.327078] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1240.331089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1240.400482] Lustre: 18589:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1240.412432] Lustre: lustre-OST0000: new disk, initializing [ 1240.414911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1240.439899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.138590] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1245.079027] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1245.706475] Lustre: 19282:0:(client.c:1533:after_reply()) @@@ resending request on EINPROGRESS req@ffff8800a78b6680 x1818499325401344/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1734256208 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 [ 1247.507540] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1247.512403] Lustre: MGS: Client 26ab654c-9508-45ae-98df-3905931f61d0 (at 0@lo) reconnecting [ 1247.527285] Lustre: Modifying parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1247.538114] LustreError: 16814:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1247.638367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1248.535459] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1248.646432] Lustre: 19811:0:(client.c:1533:after_reply()) @@@ resending request on EINPROGRESS req@ffff880098b62680 x1818499325406592/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1734256210 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 [ 1248.668419] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1254.721902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1254.726497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1257.656624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1257.678915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1277.724606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1277.727728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.734282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1277.736967] Lustre: Skipped 1 previous similar message [ 1282.358693] Lustre: server umount lustre-OST0000 complete [ 1283.949019] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.949263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1283.949373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1283.949374] Lustre: Skipped 2 previous similar messages [ 1283.961551] Lustre: Skipped 2 previous similar messages [ 1289.562822] Lustre: server umount lustre-MDT0000 complete [ 1296.900168] Lustre: server umount lustre-MDT0001 complete [ 1298.134149] Lustre: server umount MGS complete [ 1301.661913] Lustre: DEBUG MARKER: == conf-sanity-1 test 46a: handle ost additional - wide striped file ========================================================== 04:50:47 (1734256247) [ 1303.827380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1304.036449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1304.039424] Lustre: Skipped 1 previous similar message [ 1305.031164] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1307.953447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1308.007612] Lustre: 23183:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1309.247479] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1311.115382] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1312.234017] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1314.786123] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1314.794444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1316.824214] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1318.857401] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1320.279255] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1328.935980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1329.621540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 1330.698065] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1330.773151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1333.305854] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1333.312361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1333.342324] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1333.348026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1333.390057] Lustre: 26013:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1333.396979] Lustre: 26013:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 1333.411841] Lustre: lustre-OST0001: new disk, initializing [ 1333.413839] Lustre: Skipped 1 previous similar message [ 1333.416398] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1333.420500] Lustre: Skipped 2 previous similar messages [ 1333.446033] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1333.449499] Lustre: Skipped 2 previous similar messages [ 1335.256517] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1338.144274] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1340.348637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1340.352874] Lustre: Skipped 1 previous similar message [ 1340.355118] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1340.381380] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1341.269320] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1342.211845] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1342.289143] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1343.436971] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a402800.ost_server_uuid 50 [ 1343.988403] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88012a402800.ost_server_uuid in FULL state after 0 sec [ 1350.382089] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1350.382697] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.383049] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1350.383050] Lustre: Skipped 3 previous similar messages [ 1351.615322] Lustre: server umount lustre-OST0001 complete [ 1355.390382] LustreError: 24301:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.397614] LustreError: 24301:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 1358.799487] Lustre: server umount lustre-OST0000 complete [ 1360.396889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.402229] Lustre: Skipped 3 previous similar messages [ 1365.971368] Lustre: server umount lustre-MDT0000 complete [ 1367.050202] LustreError: 26027:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256313 with bad export cookie 15444839865238169869 [ 1367.051723] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1367.057114] LustreError: 26027:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1367.162735] Lustre: server umount lustre-MDT0001 complete [ 1372.755313] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 1373.398923] Key type lgssc unregistered [ 1373.485389] LNet: 29443:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1374.495238] LNet: Removed LNI 192.168.204.130@tcp [ 1383.312332] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 1387.720271] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 1387.858099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1387.898088] alg: No test for adler32 (adler32-zlib) [ 1388.738007] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1388.798197] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 1388.800458] LNet: Accept secure, port 988 [ 1390.351090] Key type lgssc registered [ 1390.585153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1393.093080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1394.967886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1396.808600] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1396.814086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1398.528589] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1398.531677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1400.558903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1400.581383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1400.589244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1401.668476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1401.679921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1401.714340] Lustre: lustre-MDT0000: new disk, initializing [ 1401.734699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1401.740813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1402.631116] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1406.676850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1406.706250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1406.741543] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1406.751291] Lustre: 2758:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1406.771486] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1406.773978] Lustre: Skipped 1 previous similar message [ 1406.810107] Lustre: lustre-MDT0001: new disk, initializing [ 1406.840985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1406.862960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1406.866979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1407.892397] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1410.899416] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1412.132491] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1414.527566] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1414.537632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1414.557487] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1414.564528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1414.661438] Lustre: 3952:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1414.674291] Lustre: lustre-OST0000: new disk, initializing [ 1414.676687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1414.711324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1416.548983] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1418.102632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1418.107296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1418.138995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1419.622684] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1420.869355] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1420.949356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1422.230507] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1422.303130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1423.134165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1423.134313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.134697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1428.143033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1428.146156] Lustre: Skipped 2 previous similar messages [ 1429.128919] Lustre: server umount lustre-OST0000 complete [ 1431.869167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1431.871317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1431.876702] Lustre: Skipped 1 previous similar message [ 1431.879752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.149007] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.693771] Lustre: server umount lustre-MDT0000 complete [ 1437.865895] LustreError: 5634:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256384 with bad export cookie 14923641310998855011 [ 1437.867119] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1437.875986] LustreError: 5634:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1437.989447] Lustre: server umount lustre-MDT0001 complete [ 1440.418072] Lustre: DEBUG MARKER: == conf-sanity-1 test 47a: server restart does not lose lru_resize ========================================================== 04:53:05 (1734256385) [ 1445.290318] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 1447.289395] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 1449.987593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1451.882531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1453.715207] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1453.721968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1455.791603] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1455.801875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1457.993239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1458.017761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1458.105009] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1458.117065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1458.159334] Lustre: lustre-MDT0000: new disk, initializing [ 1458.198362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.209589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1459.351089] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1463.825915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1463.848729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1463.885451] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1463.901284] Lustre: 10567:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1463.918991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1463.921592] Lustre: Skipped 1 previous similar message [ 1463.970878] Lustre: lustre-MDT0001: new disk, initializing [ 1464.033767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1464.056634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1464.061162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1465.472063] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1468.814523] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1470.159347] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1472.800681] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1472.806453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1472.838835] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1472.843490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1472.920532] Lustre: 11765:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1472.935057] Lustre: lustre-OST0000: new disk, initializing [ 1472.937442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1472.972439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1474.752342] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1477.937686] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1478.197648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1478.201205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1478.231238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1484.819850] Lustre: Failing over lustre-OST0000 [ 1484.880946] Lustre: server umount lustre-OST0000 complete [ 1497.148434] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1497.153577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1497.218063] Lustre: 13186:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1497.256817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.266684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.273494] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1498.570028] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1499.143445] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1499.634394] Lustre: Failing over lustre-MDT0000 [ 1499.690499] Lustre: server umount lustre-MDT0000 complete [ 1502.268680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1502.273042] Lustre: Skipped 3 previous similar messages [ 1504.255911] Lustre: 32046:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734256434/real 1734256434] req@ffff880096bd2a00 x1818499570785280/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1734256450 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 1507.276698] LustreError: 9633:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1507.284021] LustreError: 9633:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 7 previous similar messages [ 1509.171111] LustreError: 12512:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.267845] Lustre: 32046:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734256439/real 1734256439] req@ffff88012d7e4380 x1818499570787200/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1734256455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 1512.078604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1512.141203] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1512.146549] Lustre: 9631:0:(mgc_request.c:1858:mgc_process_log()) MGC192.168.204.130@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 1512.214288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 1512.241159] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1513.134721] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1514.176688] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1517.232894] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1517.239394] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1517.243241] Lustre: Skipped 1 previous similar message [ 1517.258714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 1522.237590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.243097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1522.245506] Lustre: Skipped 3 previous similar messages [ 1524.165252] Lustre: server umount lustre-OST0000 complete [ 1527.245772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.248920] Lustre: Skipped 1 previous similar message [ 1531.305827] Lustre: server umount lustre-MDT0000 complete [ 1532.252937] LustreError: 10588:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.261740] LustreError: 10588:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 1532.666390] LustreError: 9620:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256478 with bad export cookie 14923641310998857062 [ 1532.669561] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1532.674987] LustreError: 9620:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1532.843846] Lustre: server umount lustre-MDT0001 complete [ 1536.768997] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 1537.508450] Key type lgssc unregistered [ 1537.619438] LNet: 16003:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.630525] LNet: Removed LNI 192.168.204.130@tcp [ 1556.321987] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 1556.481897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1556.494221] alg: No test for adler32 (adler32-zlib) [ 1557.307471] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1557.362037] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 1557.363998] LNet: Accept secure, port 988 [ 1558.896985] Key type lgssc registered [ 1559.139458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1561.272245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1561.278032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.414899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1563.226829] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1565.900365] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1567.811994] LustreError: 17684:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1567.817910] LustreError: 17684:0:(lod_sub_object.c:914:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1567.822496] LustreError: 17684:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1567.964404] Lustre: server umount lustre-MDT0000 complete [ 1573.001569] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 1575.121166] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 1577.948658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1580.374067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1582.284596] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1582.290173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1584.402819] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1584.411282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1586.675728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1586.697337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1586.794282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1586.808060] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1586.847563] Lustre: lustre-MDT0000: new disk, initializing [ 1586.874089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.880285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1587.902856] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1592.139494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1592.165273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1592.201150] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1592.214934] Lustre: 22526:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1592.230651] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1592.233069] Lustre: Skipped 1 previous similar message [ 1592.265746] Lustre: lustre-MDT0001: new disk, initializing [ 1592.296991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1592.311147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1592.317778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1593.223651] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1595.941392] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1596.836602] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1598.654042] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1598.658351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1598.675826] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1598.680267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1598.748362] Lustre: 23723:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1598.758863] Lustre: lustre-OST0000: new disk, initializing [ 1598.761483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1598.784194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1600.236054] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1602.250325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1602.255572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1602.276651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1603.021026] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1609.692744] Lustre: DEBUG MARKER: SKIP: conf-sanity-1 test_47b Can not create symlink /usr/sbin/lctl [ 1610.151672] Lustre: DEBUG MARKER: == conf-sanity-1 test 48: too many acls on file ========== 04:55:55 (1734256555) [ 1697.420325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.420480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1697.428477] Lustre: Skipped 2 previous similar messages [ 1700.495565] Lustre: server umount lustre-MDT0000 complete [ 1701.676630] LustreError: 21582:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256647 with bad export cookie 8501344919695438730 [ 1701.677610] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1701.685232] LustreError: 21582:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1702.428168] LustreError: 25900:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1702.428222] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.428502] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1702.428503] Lustre: Skipped 2 previous similar messages [ 1702.441040] LustreError: 25900:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 1706.436217] LustreError: 25871:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.436264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1707.790512] Lustre: server umount lustre-MDT0001 complete [ 1714.944868] Lustre: server umount lustre-OST0000 complete [ 1717.286233] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 1719.443149] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 1722.509218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1724.570425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1726.373902] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1726.379907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1728.386713] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1728.393381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1730.672053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1730.697949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.776539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1730.788756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1730.824088] Lustre: lustre-MDT0000: new disk, initializing [ 1730.858602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1730.865675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1731.851821] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1735.804512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1735.843434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1735.890227] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1735.905860] Lustre: 30379:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1735.927628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1735.930749] Lustre: Skipped 1 previous similar message [ 1735.969194] Lustre: lustre-MDT0001: new disk, initializing [ 1736.020898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1736.027544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1737.128035] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1740.098314] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1741.081062] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1743.462199] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1743.470309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1743.511981] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1743.519081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1743.607178] Lustre: 31572:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1743.642927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.646822] Lustre: Skipped 1 previous similar message [ 1743.793438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1743.827047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1745.131416] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1747.912839] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1749.024874] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1749.117429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1750.152223] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1750.211624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1753.823358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1753.824214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.826219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.989703] Lustre: server umount lustre-OST0000 complete [ 1758.830176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1758.837891] Lustre: Skipped 1 previous similar message [ 1758.841275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1758.844381] Lustre: Skipped 2 previous similar messages [ 1764.509146] Lustre: server umount lustre-MDT0000 complete [ 1765.731589] LustreError: 29434:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256712 with bad export cookie 8501344919695597546 [ 1765.733361] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1765.740132] LustreError: 29434:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1765.866085] Lustre: server umount lustre-MDT0001 complete [ 1768.282932] Lustre: DEBUG MARKER: == conf-sanity-1 test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:58:33 (1734256713) [ 1773.218147] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 1775.357919] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 1778.434298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1780.561475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1782.818557] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1782.826553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1785.233555] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1785.246011] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1787.824078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1787.857720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1787.941122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1787.951473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1787.954002] Lustre: Skipped 1 previous similar message [ 1787.993193] Lustre: lustre-MDT0000: new disk, initializing [ 1787.995018] Lustre: Skipped 1 previous similar message [ 1788.024293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1788.032124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1788.035689] Lustre: Skipped 1 previous similar message [ 1789.146694] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1793.382118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1793.406356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1793.447707] Lustre: 5758:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1793.541276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1794.586717] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1797.817187] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1799.088754] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1801.711073] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1801.722204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1801.751928] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1801.759719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1801.851047] Lustre: 6955:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1803.778320] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1807.050747] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1807.570780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1807.573929] Lustre: Skipped 1 previous similar message [ 1807.578164] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1807.610084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1817.606478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1817.609006] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1817.618583] Lustre: Skipped 2 previous similar messages [ 1817.622887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.632335] Lustre: Skipped 3 previous similar messages [ 1820.195063] Lustre: server umount lustre-OST0000 complete [ 1827.933541] Lustre: server umount lustre-MDT0000 complete [ 1829.528042] LustreError: 6986:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256775 with bad export cookie 8501344919695598414 [ 1829.528430] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1829.537827] LustreError: 6986:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1829.684343] Lustre: server umount lustre-MDT0001 complete [ 1832.959002] Lustre: DEBUG MARKER: == conf-sanity-1 test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:59:38 (1734256778) [ 1838.684567] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 1840.857237] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 1844.009228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1846.393232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1848.547096] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1848.553629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1850.658414] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1850.667074] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1853.057736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1853.084842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1853.184143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1853.186863] Lustre: Skipped 1 previous similar message [ 1853.201788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1853.204306] Lustre: Skipped 3 previous similar messages [ 1853.250598] Lustre: lustre-MDT0000: new disk, initializing [ 1853.252254] Lustre: Skipped 2 previous similar messages [ 1853.283933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.286145] Lustre: Skipped 2 previous similar messages [ 1853.294466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1854.386989] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1858.508828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1858.542703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1858.588830] Lustre: 13178:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1858.679236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1859.650436] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1862.867359] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1864.147558] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1866.838933] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1866.847445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1866.876474] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1866.887008] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1867.114658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1869.080777] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1872.309768] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1877.101976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1877.105817] LustreError: Skipped 1 previous similar message [ 1877.109892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.118276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1877.118277] Lustre: Skipped 4 previous similar messages [ 1877.122594] Lustre: Skipped 3 previous similar messages [ 1880.635389] Lustre: server umount lustre-OST0000 complete [ 1883.692861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1883.695639] LustreError: Skipped 1 previous similar message [ 1888.280626] Lustre: server umount lustre-MDT0000 complete [ 1889.683766] LustreError: 13197:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256835 with bad export cookie 8501344919695599478 [ 1889.685117] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1889.692435] LustreError: 13197:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1889.824627] Lustre: server umount lustre-MDT0001 complete [ 1893.527067] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 1894.360728] Key type lgssc unregistered [ 1894.487005] LNet: 16463:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.498110] LNet: Removed LNI 192.168.204.130@tcp [ 1903.634396] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 1903.827646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1903.867741] alg: No test for adler32 (adler32-zlib) [ 1904.719661] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1904.784576] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 1904.787162] LNet: Accept secure, port 988 [ 1906.343083] Key type lgssc registered [ 1906.648824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1909.559672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1909.570235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.778752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1911.955223] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1915.040680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1915.098060] Lustre: 18838:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1915.222964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1916.442920] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1918.174666] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1919.357221] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1921.673461] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1921.681320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1921.779839] Lustre: 19948:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1921.832872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.455364] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1925.186303] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1927.645918] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1927.655077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1927.691343] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1927.697045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1927.737598] Lustre: 21068:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1927.751409] Lustre: lustre-OST0001: new disk, initializing [ 1927.755879] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1927.803754] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1929.436631] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1931.320898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1931.324937] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1931.346330] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1932.621672] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1936.338242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 1939.853898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1939.859344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.865696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1941.341439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1941.342225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1941.353838] Lustre: Skipped 1 previous similar message [ 1943.839026] Lustre: lustre-OST0000: Not available for connect from 192.168.204.30@tcp (stopping) [ 1945.576738] Lustre: server umount lustre-OST0000 complete [ 1946.353138] LustreError: 19955:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.359199] LustreError: 19955:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 1947.365574] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1947.443333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 1948.595700] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1948.665226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 1948.833635] LustreError: 21951:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1951.357889] LustreError: 19955:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1951.368847] LustreError: 19955:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 1953.844208] LustreError: 19955:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1961.374817] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.375202] LustreError: 21951:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1961.375204] LustreError: 21951:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 1961.376716] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1961.394915] Lustre: Skipped 1 previous similar message [ 1965.033200] Lustre: server umount lustre-OST0001 complete [ 1970.284623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1970.290175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1970.298439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1970.303421] Lustre: Skipped 1 previous similar message [ 1972.888969] Lustre: server umount lustre-MDT0000 complete [ 1974.426848] LustreError: 19971:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256920 with bad export cookie 189568379913664164 [ 1974.428967] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1974.439819] LustreError: 19971:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1974.573058] Lustre: server umount lustre-MDT0001 complete [ 1976.743168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1978.234162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1979.679879] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1979.687525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1981.175228] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1981.181520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1983.931016] Lustre: DEBUG MARKER: == conf-sanity-1 test 50d: lazystatfs client/server conn race ========================================================== 05:02:09 (1734256929) [ 1986.720677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.844337] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1986.862129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1986.955345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.262976] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1991.768849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1991.829314] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1991.855334] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1991.872219] Lustre: 26128:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1993.372684] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1995.599968] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1997.041164] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2000.554334] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2000.566586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2000.658880] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2000.687525] Lustre: 27251:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2000.781333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2000.786007] Lustre: Skipped 1 previous similar message [ 2002.196374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [ 2003.186399] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2005.518927] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2008.590560] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2008.603403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2008.649515] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2008.692704] Lustre: 28387:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2011.054460] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2012.138091] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2013.290033] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2017.152286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.157978] Lustre: Skipped 1 previous similar message [ 2017.160195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2017.162892] Lustre: Skipped 2 previous similar messages [ 2020.549007] Lustre: server umount lustre-OST0000 complete [ 2022.159442] LustreError: 27490:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2022.181711] LustreError: 27490:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 2032.142584] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2032.147365] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.152516] Lustre: Skipped 1 previous similar message [ 2037.181725] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2037.186893] Lustre: Skipped 5 previous similar messages [ 2037.997176] Lustre: server umount lustre-OST0001 complete [ 2042.060669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2046.019273] Lustre: server umount lustre-MDT0000 complete [ 2047.196555] LustreError: 25264:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.208383] LustreError: 25264:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 9 previous similar messages [ 2047.467387] LustreError: 25246:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256993 with bad export cookie 189568379913665557 [ 2047.471298] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2047.482146] LustreError: 25246:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2047.652616] Lustre: server umount lustre-MDT0001 complete [ 2049.859127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2051.531653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2053.320127] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2053.329994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2054.940881] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2054.949376] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2057.571845] Lustre: DEBUG MARKER: == conf-sanity-1 test 50f: normal statfs one server in down ========================================================== 05:03:22 (1734257002) [ 2060.360343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.500874] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2060.520454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2060.609323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2060.611995] Lustre: Skipped 1 previous similar message [ 2061.876061] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2065.498018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2065.560148] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2065.582712] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2065.597199] Lustre: 375:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2066.761455] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2068.530230] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2069.891469] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2072.463043] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2072.470273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2072.588702] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2074.783221] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2076.812840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2076.850720] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2078.093743] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2078.157422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2079.254282] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2079.320137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2081.845515] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2081.854164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2081.895769] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2082.264704] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 2083.866902] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2085.634633] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2086.924700] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2087.013293] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2088.352176] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2088.442698] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2092.254937] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2092.259551] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.266219] Lustre: Skipped 3 previous similar messages [ 2092.271638] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2092.273719] Lustre: Skipped 2 previous similar messages [ 2095.263323] Lustre: server umount lustre-OST0001 complete [ 2097.131270] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2097.228050] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2097.297143] LustreError: 2202:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2097.302303] LustreError: 2202:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 2098.518355] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2098.600964] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2122.058628] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2122.066940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2122.110341] Lustre: 5680:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2122.114871] Lustre: 5680:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 2 previous similar messages [ 2122.178079] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2122.181277] Lustre: Skipped 3 previous similar messages [ 2123.857043] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2125.579062] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2126.210688] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2132.919221] Lustre: server umount lustre-OST0001 complete [ 2146.909092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2146.912236] LustreError: Skipped 1 previous similar message [ 2146.913620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.918782] Lustre: Skipped 1 previous similar message [ 2150.629849] Lustre: server umount lustre-OST0000 complete [ 2157.244358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2157.247266] Lustre: Skipped 13 previous similar messages [ 2157.746574] Lustre: server umount lustre-MDT0000 complete [ 2159.066488] LustreError: 392:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257105 with bad export cookie 189568379913667328 [ 2159.067461] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2159.075904] LustreError: 392:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2165.249075] Lustre: server umount lustre-MDT0001 complete [ 2167.541870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2169.273695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2171.150073] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2171.161833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2172.677852] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2172.686517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2175.190427] Lustre: DEBUG MARKER: == conf-sanity-1 test 50g: deactivated OST should not cause panic ========================================================== 05:05:20 (1734257120) [ 2177.498181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.611716] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2177.624595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2178.741950] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2181.300180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2181.361904] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2182.550082] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2184.268197] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2185.373475] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2188.334859] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2188.347254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2188.474848] Lustre: 11307:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2188.480098] Lustre: 11307:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2188.540478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.542674] Lustre: Skipped 2 previous similar messages [ 2190.722512] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2192.880828] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2195.021829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 2202.021852] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2202.036077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2202.565607] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 2204.182111] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2206.317381] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2210.997415] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 2212.063624] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2212.128763] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2213.249583] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2213.309192] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2214.502550] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800abe7f000.ost_server_uuid 50 [ 2215.033305] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8800abe7f000.ost_server_uuid in FULL state after 0 sec [ 2215.373430] Lustre: Permanently deactivating lustre-OST0001 [ 2215.379767] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 2215.382307] Lustre: Skipped 1 previous similar message [ 2216.132391] Lustre: Permanently reactivating lustre-OST0001 [ 2217.565168] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2217.567661] LustreError: Skipped 1 previous similar message [ 2217.569069] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.574106] Lustre: Skipped 3 previous similar messages [ 2222.909333] Lustre: server umount lustre-OST0001 complete [ 2227.597381] LustreError: 11314:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.603802] LustreError: 11314:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 21 previous similar messages [ 2230.128513] Lustre: server umount lustre-OST0000 complete [ 2237.286133] Lustre: server umount lustre-MDT0000 complete [ 2238.341550] LustreError: 9298:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257184 with bad export cookie 189568379913669190 [ 2238.342640] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2238.351143] LustreError: 9298:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2238.458570] Lustre: server umount lustre-MDT0001 complete [ 2240.069282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2241.236369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2242.374942] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2242.382216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2243.665340] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2243.671266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2245.538200] Lustre: DEBUG MARKER: == conf-sanity-1 test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:06:30 (1734257190) [ 2250.455198] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 2252.494598] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2255.522875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2258.049606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2260.273219] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2260.283260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2262.608427] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2262.617327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2265.498006] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2265.531302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.634882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2265.637420] Lustre: Skipped 5 previous similar messages [ 2265.656592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2265.701683] Lustre: lustre-MDT0000: new disk, initializing [ 2265.746857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2266.949120] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2271.659103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2271.705682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2271.765686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2271.768181] Lustre: Skipped 1 previous similar message [ 2271.805893] Lustre: lustre-MDT0001: new disk, initializing [ 2271.863391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2271.868938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2273.259830] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2276.418280] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2277.682482] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2280.155685] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2280.162302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2280.186624] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2280.192909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2280.302652] Lustre: lustre-OST0000: new disk, initializing [ 2280.309002] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2282.229702] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2285.304672] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2286.254864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2286.261380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2286.299231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2291.051057] LustreError: 19829:0:(mdt_handler.c:3146:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 2295.068585] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2295.071855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2295.092804] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2295.097375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2295.143285] Lustre: lustre-OST0001: new disk, initializing [ 2295.145728] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2296.593826] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2299.132187] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2300.885635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2300.889162] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2300.907480] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2301.062902] LustreError: 19829:0:(mdt_handler.c:3146:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 2301.070692] LustreError: 19829:0:(mdt_handler.c:3146:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 2301.578561] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 2311.076068] LustreError: 19829:0:(mdt_handler.c:3146:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 2315.670832] Lustre: lustre-OST0001: Not available for connect from 192.168.204.30@tcp (stopping) [ 2315.673500] Lustre: Skipped 11 previous similar messages [ 2315.916996] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2315.920273] LustreError: Skipped 4 previous similar messages [ 2317.686713] Lustre: server umount lustre-OST0001 complete [ 2334.998302] Lustre: server umount lustre-OST0000 complete [ 2342.042000] Lustre: server umount lustre-MDT0000 complete [ 2343.199248] LustreError: 23247:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257289 with bad export cookie 189568379913671262 [ 2343.201720] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2343.208596] LustreError: 23247:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2349.314170] Lustre: server umount lustre-MDT0001 complete [ 2352.733205] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 2353.386144] Key type lgssc unregistered [ 2353.482564] LNet: 25539:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2353.495292] LNet: Removed LNI 192.168.204.130@tcp [ 2364.312398] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 2369.173457] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2369.324552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2369.359518] alg: No test for adler32 (adler32-zlib) [ 2370.197803] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 2370.261072] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 2370.264767] LNet: Accept secure, port 988 [ 2371.809367] Key type lgssc registered [ 2372.104062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2375.450015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2378.190725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2380.264634] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2380.270105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2382.225550] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2382.232393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2384.481640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2384.501088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2384.507769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.584613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2385.605104] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2385.645267] Lustre: lustre-MDT0000: new disk, initializing [ 2385.675848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2385.686676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2386.575816] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2390.352693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2390.371280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2390.401329] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2390.410406] Lustre: 31282:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2390.426917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2390.429642] Lustre: Skipped 1 previous similar message [ 2390.462578] Lustre: lustre-MDT0001: new disk, initializing [ 2390.488542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2390.501327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2390.505417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2391.380165] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2394.007075] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2394.856032] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2396.662642] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2396.667167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2396.690821] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2396.694550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2396.768155] Lustre: 32478:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2396.779662] Lustre: lustre-OST0000: new disk, initializing [ 2396.781706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2396.805250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.104730] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2400.751999] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2401.680584] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2403.814273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2403.817707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2403.831503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2404.781824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2405.601846] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2405.654140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2408.828823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2408.828978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2408.829192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2408.838145] Lustre: Skipped 1 previous similar message [ 2412.208816] Lustre: server umount lustre-OST0000 complete [ 2413.836421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.840490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2413.841901] Lustre: Skipped 2 previous similar messages [ 2418.844351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2418.846343] Lustre: Skipped 1 previous similar message [ 2419.284686] Lustre: server umount lustre-MDT0000 complete [ 2420.260645] LustreError: 31296:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257366 with bad export cookie 3128566906844821119 [ 2420.262114] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2420.267852] LustreError: 31296:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2420.371556] Lustre: server umount lustre-MDT0001 complete [ 2422.448075] Lustre: DEBUG MARKER: == conf-sanity-1 test 54a: test llverdev and partial verify of device ========================================================== 05:09:27 (1734257367) [ 2423.036817] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 2427.732575] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 2429.399540] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2432.328984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2434.560752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2436.594780] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2436.600610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2438.858951] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2438.871253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2441.316499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2441.348291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2441.467234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2441.492021] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2441.545183] Lustre: lustre-MDT0000: new disk, initializing [ 2441.593715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.607001] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2442.744465] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2447.329716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2447.355991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2447.387983] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2447.397743] Lustre: 6864:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2447.415943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2447.419352] Lustre: Skipped 1 previous similar message [ 2447.461283] Lustre: lustre-MDT0001: new disk, initializing [ 2447.497134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2447.518742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2447.523686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2448.610486] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2451.833340] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2453.021936] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2455.165679] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2455.175373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2455.201577] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2455.205512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2455.276645] Lustre: 8060:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2455.463011] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2455.491037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2456.846422] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2459.625927] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2460.523863] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2460.578714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2461.479008] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2461.545939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2465.500627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2465.500838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2465.501218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2465.511021] Lustre: Skipped 2 previous similar messages [ 2468.261821] Lustre: server umount lustre-OST0000 complete [ 2470.508463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.508835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2470.508836] Lustre: Skipped 1 previous similar message [ 2470.517392] Lustre: Skipped 1 previous similar message [ 2475.423036] Lustre: server umount lustre-MDT0000 complete [ 2476.392572] LustreError: 9738:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257422 with bad export cookie 3128566906844821987 [ 2476.393845] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2476.401265] LustreError: 9738:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2476.495948] Lustre: server umount lustre-MDT0001 complete [ 2478.613228] Lustre: DEBUG MARKER: == conf-sanity-1 test 55: check lov_objid size =========== 05:10:24 (1734257424) [ 2480.055250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2481.285262] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2481.288099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2483.014662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2483.032812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2483.106307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2483.120988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2483.123982] Lustre: Skipped 1 previous similar message [ 2483.157769] Lustre: lustre-MDT0000: new disk, initializing [ 2483.160422] Lustre: Skipped 1 previous similar message [ 2483.193743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.196300] Lustre: Skipped 1 previous similar message [ 2483.203531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2483.206493] Lustre: Skipped 1 previous similar message [ 2484.077164] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2487.866527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2487.904339] LustreError: lustre-MDT0001 claims to have registered, but this MGS does not know about it, preventing registration. [ 2487.909120] LustreError: lustre-MDT0001: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 2487.913540] LustreError: 12377:0:(tgt_mount.c:2222:server_fill_super()) Unable to start targets: -2 [ 2487.916370] LustreError: 12377:0:(tgt_mount.c:1721:server_put_super()) no obd lustre-MDT0001 [ 2487.919720] LustreError: 12377:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0001 not registered [ 2487.925047] Lustre: server umount lustre-MDT0001 complete [ 2487.927469] LustreError: 12377:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2489.836592] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2489.841843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2489.864202] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2489.867917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2489.953890] Lustre: 12840:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2491.020708] Lustre: cli-lustre-OST03ff-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:3ff:ost] [ 2491.034994] Lustre: lustre-OST03ff-osc-MDT0000: update sequence from 0x103ff0000 to 0x240000400 [ 2491.501020] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2493.180405] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03ff-osc-[-0-9a-f]*.ost_server_uuid [ 2496.012438] Lustre: lustre-MDT0000-lwp-OST03ff: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2496.017222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2496.019624] Lustre: Skipped 1 previous similar message [ 2501.344163] Lustre: server umount lustre-MDT0000 complete [ 2503.537193] LustreError: 11432:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257449 with bad export cookie 3128566906844822855 [ 2503.538474] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2503.545725] LustreError: 11432:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2503.610761] Lustre: server umount lustre-OST03ff complete [ 2507.138073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.331346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.334304] Lustre: Skipped 1 previous similar message [ 2508.419987] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2511.444606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2511.495680] LustreError: lustre-MDT0001 claims to have registered, but this MGS does not know about it, preventing registration. [ 2511.500876] LustreError: lustre-MDT0001: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 2511.506301] LustreError: 15798:0:(tgt_mount.c:2222:server_fill_super()) Unable to start targets: -2 [ 2511.510341] LustreError: 15798:0:(tgt_mount.c:1721:server_put_super()) no obd lustre-MDT0001 [ 2511.513721] LustreError: 15798:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0001 not registered [ 2511.521843] Lustre: server umount lustre-MDT0001 complete [ 2511.523824] LustreError: 15798:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2513.497685] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2513.502410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2513.566730] Lustre: 16259:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2515.045425] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2516.486381] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03ff-osc-[-0-9a-f]*.ost_server_uuid [ 2527.612578] Lustre: lustre-MDT0000-lwp-OST03ff: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.618940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.621523] Lustre: Skipped 1 previous similar message [ 2528.732930] Lustre: server umount lustre-MDT0000 complete [ 2531.000002] LustreError: 14920:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257477 with bad export cookie 3128566906844823401 [ 2531.001228] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2531.010348] LustreError: 14920:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2531.085327] Lustre: server umount lustre-OST03ff complete [ 2534.172466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2535.753442] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2535.760098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2538.001379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2538.025718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.104227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2538.118175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2538.120517] Lustre: Skipped 2 previous similar messages [ 2538.154106] Lustre: lustre-MDT0000: new disk, initializing [ 2538.155924] Lustre: Skipped 1 previous similar message [ 2538.201345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2538.205806] Lustre: Skipped 1 previous similar message [ 2539.208836] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2543.118164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2543.172388] LustreError: lustre-MDT0001 claims to have registered, but this MGS does not know about it, preventing registration. [ 2543.178682] LustreError: lustre-MDT0001: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 2543.182179] LustreError: 19904:0:(tgt_mount.c:2222:server_fill_super()) Unable to start targets: -2 [ 2543.185129] LustreError: 19904:0:(tgt_mount.c:1721:server_put_super()) no obd lustre-MDT0001 [ 2543.187834] LustreError: 19904:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0001 not registered [ 2543.194333] Lustre: server umount lustre-MDT0001 complete [ 2543.196768] LustreError: 19904:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2545.317111] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2545.328123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2545.365670] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2545.374108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2545.474173] Lustre: 20367:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2545.514605] Lustre: lustre-OST0800: Imperative Recovery not enabled, recovery window 60-180 [ 2545.517815] Lustre: Skipped 2 previous similar messages [ 2546.391046] Lustre: cli-lustre-OST0800-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:800:ost] [ 2546.405881] Lustre: lustre-OST0800-osc-MDT0000: update sequence from 0x108000000 to 0x240000400 [ 2547.119852] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2550.031854] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0800-osc-[-0-9a-f]*.ost_server_uuid [ 2551.388529] Lustre: lustre-MDT0000-lwp-OST0800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.001191] Lustre: server umount lustre-MDT0000 complete [ 2559.006662] LustreError: 21235:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257505 with bad export cookie 3128566906844823891 [ 2559.010407] LustreError: 21235:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2559.012576] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2559.074227] Lustre: server umount lustre-OST0800 complete [ 2562.562126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.769489] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2566.672931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2566.723720] LustreError: lustre-MDT0001 claims to have registered, but this MGS does not know about it, preventing registration. [ 2566.727063] LustreError: lustre-MDT0001: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 2566.733908] LustreError: 23387:0:(tgt_mount.c:2222:server_fill_super()) Unable to start targets: -2 [ 2566.736528] LustreError: 23387:0:(tgt_mount.c:1721:server_put_super()) no obd lustre-MDT0001 [ 2566.739382] LustreError: 23387:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0001 not registered [ 2566.744523] Lustre: server umount lustre-MDT0001 complete [ 2566.746248] LustreError: 23387:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2568.917913] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2568.922906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2570.559670] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2571.968426] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0800-osc-[-0-9a-f]*.ost_server_uuid [ 2578.044939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2578.046906] Lustre: Skipped 2 previous similar messages [ 2584.138537] Lustre: server umount lustre-MDT0000 complete [ 2586.606958] LustreError: 24599:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257532 with bad export cookie 3128566906844824437 [ 2586.607499] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2586.622225] LustreError: 24599:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2586.686083] Lustre: server umount lustre-OST0800 complete [ 2593.588982] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 2595.585336] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2598.440370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2600.425743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2602.241837] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2602.245788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2604.027919] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2604.033768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2606.161315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2606.195571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.293334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2606.304676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2606.307419] Lustre: Skipped 2 previous similar messages [ 2606.344537] Lustre: lustre-MDT0000: new disk, initializing [ 2606.346433] Lustre: Skipped 1 previous similar message [ 2606.379690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2606.384205] Lustre: Skipped 1 previous similar message [ 2607.427442] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2611.639768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2611.670829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2611.711419] Lustre: 29792:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2611.715978] Lustre: 29792:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2611.786990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2611.790145] Lustre: Skipped 3 previous similar messages [ 2611.805192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2612.831896] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2615.764157] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2616.797619] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2619.271646] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2619.279930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2619.311937] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2619.319968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2621.454670] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2624.525175] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2625.722223] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2625.768211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2626.837495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2627.968561] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2628.033423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2630.764974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2630.765272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2630.765274] Lustre: Skipped 1 previous similar message [ 2630.775992] LustreError: Skipped 2 previous similar messages [ 2634.772168] Lustre: server umount lustre-OST0000 complete [ 2636.814006] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2642.268259] Lustre: server umount lustre-MDT0000 complete [ 2643.479231] LustreError: 28847:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257589 with bad export cookie 3128566906844824927 [ 2643.481047] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2643.487387] LustreError: 28847:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2643.592603] Lustre: server umount lustre-MDT0001 complete [ 2646.626547] Lustre: DEBUG MARKER: == conf-sanity-1 test 56a: check big OST indexes and out-of-index-order start ========================================================== 05:13:11 (1734257591) [ 2651.917517] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 2654.021875] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2656.999570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2659.485726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2661.783549] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2661.794295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2664.267854] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2664.272664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2665.980066] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2665.987315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2667.614956] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2667.623519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2670.240633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2670.281438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2670.395974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2670.399005] Lustre: Skipped 1 previous similar message [ 2670.414954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2670.417957] Lustre: Skipped 3 previous similar messages [ 2670.460447] Lustre: lustre-MDT0000: new disk, initializing [ 2670.462303] Lustre: Skipped 2 previous similar messages [ 2670.516764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2670.520491] Lustre: Skipped 2 previous similar messages [ 2671.866198] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2676.939510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2676.976301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2677.025967] Lustre: 5743:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2677.034009] Lustre: 5743:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2677.174216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2677.178194] Lustre: Skipped 1 previous similar message [ 2678.497445] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2682.017482] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2683.578338] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2686.966426] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2686.973655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2687.013995] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2687.022141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2689.361745] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2692.943571] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 2694.614376] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 2695.626446] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2695.632589] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2695.655486] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2695.662529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2697.844478] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2699.955349] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 2699.960101] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 2701.694416] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 2701.778535] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2703.211596] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 2703.322716] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2704.555132] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 2704.633204] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2705.797144] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 2705.880278] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2707.182221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2707.190055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2707.195819] Lustre: Skipped 3 previous similar messages [ 2707.201838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2707.207979] Lustre: Skipped 6 previous similar messages [ 2713.121510] Lustre: server umount lustre-MDT0000 complete [ 2714.521972] LustreError: 4797:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257660 with bad export cookie 3128566906844825795 [ 2714.523358] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2714.529767] LustreError: 4797:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2714.677170] Lustre: server umount lustre-MDT0001 complete [ 2726.018889] Lustre: server umount lustre-OST2710 complete [ 2737.616428] Lustre: server umount lustre-OST03e8 complete [ 2743.808274] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 2745.983183] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2749.060328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2751.347913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2753.544603] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2753.551023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2755.674601] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2755.679460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2758.103650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2758.135866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2758.233690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2758.237554] Lustre: Skipped 1 previous similar message [ 2758.332492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2758.335547] Lustre: Skipped 5 previous similar messages [ 2759.420741] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2764.074405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2764.105659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2764.257069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2764.260024] Lustre: Skipped 2 previous similar messages [ 2765.581354] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2768.879928] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2770.217169] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2772.673357] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2772.680829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2772.705124] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2772.710228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2774.461732] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2775.616906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2777.474119] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2778.633246] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2778.743989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2779.864622] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2779.935874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2780.621448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2786.690895] Lustre: server umount lustre-OST0000 complete [ 2794.065493] Lustre: server umount lustre-MDT0000 complete [ 2795.491459] LustreError: 16446:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257741 with bad export cookie 3128566906844827461 [ 2795.496904] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2795.669225] Lustre: server umount lustre-MDT0001 complete [ 2799.052460] Lustre: DEBUG MARKER: == conf-sanity-1 test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:15:44 (1734257744) [ 2799.776256] Lustre: DEBUG MARKER: SKIP: conf-sanity-1 test_56b needs >= 3 MDTs [ 2800.544981] Lustre: DEBUG MARKER: == conf-sanity-1 test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:15:45 (1734257745) [ 2801.454442] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2804.100311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2805.472163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2806.842387] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2806.849836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2808.299195] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2808.306084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2808.767388] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2808.772443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2811.146295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2811.266884] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2812.454952] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2815.567305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2815.623423] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2815.648089] Lustre: 21885:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2815.652491] Lustre: 21885:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 4 previous similar messages [ 2817.051090] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2819.117005] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2820.442880] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2823.543573] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2823.552818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2823.592028] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2823.603316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2823.702815] Lustre: Denying initial registration attempt from nid 192.168.204.130@tcp, specified as failover [ 2823.712075] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 2823.718344] LustreError: 23011:0:(tgt_mount.c:2222:server_fill_super()) Unable to start targets: -99 [ 2823.722938] LustreError: 23011:0:(tgt_mount.c:1721:server_put_super()) no obd lustre-OST0000 [ 2823.727408] LustreError: 23011:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2823.797121] Lustre: server umount lustre-OST0000 complete [ 2823.799038] LustreError: 23011:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 2832.547116] Lustre: server umount lustre-MDT0000 complete [ 2833.036479] LustreError: 22315:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.042194] LustreError: 22315:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 4 previous similar messages [ 2833.849927] Lustre: server umount lustre-MDT0001 complete [ 2837.512569] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 2838.236935] Key type lgssc unregistered [ 2838.322167] LNet: 24301:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2839.328639] LNet: Removed LNI 192.168.204.130@tcp [ 2842.190080] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2842.342869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2842.348055] alg: No test for adler32 (adler32-zlib) [ 2843.185390] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 2843.261989] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 2843.264606] LNet: Accept secure, port 988 [ 2844.813871] Key type lgssc registered [ 2845.062155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2847.241224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2848.522926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2849.745892] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2849.754996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2851.136060] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2851.141859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2851.526842] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2851.532994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2857.481029] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2860.571004] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2860.581392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.691767] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2861.702280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2861.771937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.964659] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2866.396947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2866.462096] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2866.484750] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2866.500380] Lustre: 28318:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2866.599783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2867.728231] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2869.881704] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2871.164124] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2873.847940] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2873.854399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2873.892820] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2873.899033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2874.000827] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2874.022946] Lustre: 29440:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2874.078177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2875.855492] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2877.660960] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2878.707332] Lustre: server umount lustre-OST0000 complete [ 2881.598391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2881.602764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2881.616949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2882.477779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2882.478098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.487720] Lustre: Skipped 1 previous similar message [ 2886.470116] Lustre: server umount lustre-MDT0000 complete [ 2887.916105] LustreError: 28329:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257834 with bad export cookie 5193443986928763575 [ 2887.918398] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2887.932811] LustreError: 28329:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2888.079550] Lustre: server umount lustre-MDT0001 complete [ 2891.110256] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 2892.088032] Key type lgssc unregistered [ 2892.224551] LNet: 31311:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2893.231213] LNet: Removed LNI 192.168.204.130@tcp [ 2900.594738] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2900.720235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2900.756286] alg: No test for adler32 (adler32-zlib) [ 2901.603039] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 2901.657557] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 2901.659686] LNet: Accept secure, port 988 [ 2903.206113] Key type lgssc registered [ 2903.529246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2906.436999] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2906.448319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.699290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.872958] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2911.676597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2911.727171] Lustre: 1259:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2911.810211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2912.688295] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2914.294564] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2915.290700] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2916.213612] LustreError: 390:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2916.219438] LustreError: 390:0:(lod_sub_object.c:914:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2916.222929] LustreError: 390:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 2916.798133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2916.804162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.811558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2916.813520] Lustre: Skipped 1 previous similar message [ 2921.814471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.817585] Lustre: Skipped 1 previous similar message [ 2922.285618] Lustre: server umount lustre-MDT0000 complete [ 2923.650376] LustreError: 2107:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257869 with bad export cookie 14611413599914273899 [ 2923.653058] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2923.666421] LustreError: 2107:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2923.788724] Lustre: server umount lustre-MDT0001 complete [ 2928.839087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.018304] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2929.029198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2929.121625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2930.194731] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2933.364737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2933.415673] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2933.434775] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2933.447469] Lustre: 4258:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2933.547474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2934.799115] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2936.871994] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2938.228704] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2941.381672] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2941.391615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2941.508193] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 2941.516387] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 2941.523339] LustreError: 5458:0:(tgt_mount.c:2222:server_fill_super()) Unable to start targets: -2 [ 2941.525755] LustreError: 5458:0:(tgt_mount.c:1721:server_put_super()) no obd lustre-OST0000 [ 2941.527869] LustreError: 5458:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2941.598091] Lustre: server umount lustre-OST0000 complete [ 2941.599518] LustreError: 5458:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2944.737423] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2944.745925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2944.843201] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2944.862297] Lustre: 6063:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2944.935780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.907739] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2948.749937] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2951.522570] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2951.527939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2951.552807] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2951.558055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2951.590453] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2951.617033] Lustre: 7257:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2951.630904] Lustre: lustre-OST0001: new disk, initializing [ 2951.633718] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2953.489477] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2955.397292] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2956.474063] Lustre: server umount lustre-OST0001 complete [ 2956.689926] LustreError: 6071:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.704327] LustreError: 6071:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 2960.765095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2960.768222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.774295] Lustre: Skipped 1 previous similar message [ 2960.777022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.729100] LustreError: 6070:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.738964] LustreError: 6070:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 2964.142991] Lustre: server umount lustre-OST0000 complete [ 2966.734338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2966.736018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2966.736020] Lustre: Skipped 3 previous similar messages [ 2966.746368] Lustre: Skipped 2 previous similar messages [ 2971.671348] Lustre: server umount lustre-MDT0000 complete [ 2971.740532] LustreError: 3398:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.747428] LustreError: 3398:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 2973.122649] LustreError: 7269:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257919 with bad export cookie 14611413599914274431 [ 2973.125840] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2973.138535] LustreError: 7269:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2973.307681] Lustre: server umount lustre-MDT0001 complete [ 2976.144144] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 2977.106716] Key type lgssc unregistered [ 2977.218273] LNet: 9463:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2978.233201] LNet: Removed LNI 192.168.204.130@tcp [ 2991.075516] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 2997.149121] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2997.341703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2997.366460] alg: No test for adler32 (adler32-zlib) [ 2998.239593] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 2998.332896] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 2998.334745] LNet: Accept secure, port 988 [ 2999.915518] Key type lgssc registered [ 3000.332327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3003.691816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3006.232685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3008.427701] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3008.433710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3010.751037] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3010.756426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3013.235065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3013.260714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3013.270295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.369109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3014.385047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3014.422296] Lustre: lustre-MDT0000: new disk, initializing [ 3014.459973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.471885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3015.678888] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3020.641358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3020.673421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3020.723914] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3020.739920] Lustre: 15214:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3020.764928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3020.769136] Lustre: Skipped 1 previous similar message [ 3020.835303] Lustre: lustre-MDT0001: new disk, initializing [ 3020.894173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3020.932617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3020.945129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3022.385068] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3026.046479] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3027.353928] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3030.159830] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3030.165608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3030.214599] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3030.224078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3030.343517] Lustre: 16409:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3030.372848] Lustre: lustre-OST0000: new disk, initializing [ 3030.379225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3030.430064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.795094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3031.801899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3031.846020] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3032.380463] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3035.659813] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3036.804299] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3036.897898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3038.018223] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3038.083183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3041.837081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3041.840931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.848186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3044.853187] Lustre: server umount lustre-OST0000 complete [ 3046.879556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3046.901486] Lustre: Skipped 1 previous similar message [ 3046.907267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3046.910225] Lustre: Skipped 1 previous similar message [ 3050.910153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3050.913164] LustreError: Skipped 1 previous similar message [ 3050.915717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3050.921903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3052.506943] Lustre: server umount lustre-MDT0000 complete [ 3053.842197] LustreError: 14266:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734258000 with bad export cookie 9755551600786350297 [ 3053.844419] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3053.851330] LustreError: 14266:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3053.978555] Lustre: server umount lustre-MDT0001 complete [ 3056.849236] Lustre: DEBUG MARKER: == conf-sanity-1 test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 05:20:02 (1734258002) [ 3059.277468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3061.889381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3071.356005] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 3074.070192] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3077.459813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3079.778243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3081.899527] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3081.911911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3084.550118] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3084.563263] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3087.580289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3087.614700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.722504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3087.737258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3087.780582] Lustre: lustre-MDT0000: new disk, initializing [ 3087.821126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.832390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3089.200004] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3094.602317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3094.652102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3094.720970] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3094.735594] Lustre: 24043:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3094.757586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3094.759136] Lustre: Skipped 1 previous similar message [ 3094.805624] Lustre: lustre-MDT0001: new disk, initializing [ 3094.868416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3094.895146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3094.902670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3096.251663] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3099.433336] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3100.533649] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3102.740780] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3102.747057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3102.775391] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3102.779472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3102.864115] Lustre: 25239:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3102.882116] Lustre: lustre-OST0000: new disk, initializing [ 3102.883684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3102.912948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3104.534781] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3107.849105] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3109.119732] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3111.689942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3111.695239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3111.723087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3112.307485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3113.487963] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3113.554826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3116.717486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3116.718372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.718701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3116.718703] Lustre: Skipped 1 previous similar message [ 3116.731219] LustreError: Skipped 1 previous similar message [ 3120.257602] Lustre: server umount lustre-OST0000 complete [ 3121.724664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3121.730872] Lustre: Skipped 1 previous similar message [ 3121.731631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3121.731633] Lustre: Skipped 1 previous similar message [ 3127.511513] Lustre: server umount lustre-MDT0000 complete [ 3128.706869] LustreError: 25271:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734258075 with bad export cookie 9755551600786351165 [ 3128.708250] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3128.716260] LustreError: 25271:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3128.829534] Lustre: server umount lustre-MDT0001 complete [ 3131.440755] Lustre: DEBUG MARKER: == conf-sanity-1 test 62: start with disabled journal ==== 05:21:16 (1734258076) [ 3133.584564] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3133.587303] LustreError: 28131:0:(osd_handler.c:8403:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 3133.591287] LustreError: 28131:0:(obd_config.c:797:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 3133.593486] LustreError: 28131:0:(obd_mount.c:214:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 3133.596379] LustreError: 28131:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 3133.599990] LustreError: 28131:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3135.758364] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3135.761907] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3135.765258] LustreError: 28658:0:(osd_handler.c:8403:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 3135.771216] LustreError: 28658:0:(obd_config.c:797:class_setup()) setup lustre-OST0000-osd failed (-22) [ 3135.775499] LustreError: 28658:0:(obd_mount.c:214:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 3135.778016] LustreError: 28658:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 3135.781298] LustreError: 28658:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3140.524273] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 3141.240627] Key type lgssc unregistered [ 3141.329242] LNet: 29792:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3142.335284] LNet: Removed LNI 192.168.204.130@tcp [ 3148.023767] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 3153.254291] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3153.387723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3153.395163] alg: No test for adler32 (adler32-zlib) [ 3154.244092] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 3154.309666] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 3154.313337] LNet: Accept secure, port 988 [ 3155.851929] Key type lgssc registered [ 3156.222193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3159.231756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3161.293273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3163.125627] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3163.131115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3165.088410] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3165.095208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3167.399742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3167.425265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3167.433710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3168.527596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3168.538119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3168.572063] Lustre: lustre-MDT0000: new disk, initializing [ 3168.600186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.609045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3169.520944] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3173.728568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3173.753503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3173.803136] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3173.813002] Lustre: 2036:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3173.827577] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3173.830530] Lustre: Skipped 1 previous similar message [ 3173.874661] Lustre: lustre-MDT0001: new disk, initializing [ 3173.914085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3173.931018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3173.935088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3174.984438] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3177.972731] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3179.069775] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3181.118059] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3181.124184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3181.149622] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3181.154213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3181.238552] Lustre: 3231:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3181.256168] Lustre: lustre-OST0000: new disk, initializing [ 3181.257842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3181.290452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3181.587974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3181.592534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3181.633603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3182.897279] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3185.995770] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3187.115200] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3187.173437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3188.262934] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3188.339855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3191.629529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.630079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3191.631318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3191.640425] Lustre: Skipped 1 previous similar message [ 3195.023494] Lustre: server umount lustre-OST0000 complete [ 3196.638989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3196.642311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3196.642312] Lustre: Skipped 1 previous similar message [ 3196.653108] Lustre: Skipped 1 previous similar message [ 3201.645764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3201.651094] Lustre: Skipped 2 previous similar messages [ 3202.379078] Lustre: server umount lustre-MDT0000 complete [ 3204.205722] LustreError: 1085:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734258150 with bad export cookie 3002189174037472723 [ 3204.211907] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3204.226882] LustreError: 1085:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3204.385237] Lustre: server umount lustre-MDT0001 complete [ 3208.131643] Lustre: DEBUG MARKER: == conf-sanity-1 test 64: check lfs df --lazy ============ 05:22:33 (1734258153) [ 3211.234681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3211.509243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.832723] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3216.515350] LustreError: 6062:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.525748] LustreError: 6062:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 3216.558861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3216.616319] Lustre: 6923:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3216.749578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3218.098122] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3220.203615] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3221.801057] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3224.621872] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3224.630229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3224.726382] Lustre: 8038:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3224.798298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3227.139661] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3229.071805] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3232.016098] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3232.027914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3232.060566] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3232.066095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3232.112001] Lustre: 9162:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3232.126057] Lustre: lustre-OST0001: new disk, initializing [ 3232.129669] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3233.482726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3233.489306] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3233.527712] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3234.282539] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3237.667064] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3246.531840] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.533397] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3246.542028] Lustre: Skipped 1 previous similar message [ 3250.581408] Lustre: server umount lustre-OST0001 complete [ 3251.533065] LustreError: 8044:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.543542] LustreError: 8044:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 3256.542469] LustreError: 9927:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3256.549426] LustreError: 9927:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 3261.554097] LustreError: 8043:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3261.563153] LustreError: 8043:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 3266.556890] LustreError: 8335:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3266.560706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.561237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3266.561239] Lustre: Skipped 2 previous similar messages [ 3266.580849] LustreError: 8335:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 3268.816271] Lustre: server umount lustre-OST0000 complete [ 3271.566258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3271.571044] Lustre: Skipped 2 previous similar messages [ 3276.400621] Lustre: server umount lustre-MDT0000 complete [ 3276.573840] LustreError: 6062:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.589151] LustreError: 6062:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 3278.081178] LustreError: 9176:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734258224 with bad export cookie 3002189174037473598 [ 3278.081300] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3278.093236] LustreError: 9176:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3284.205138] Lustre: server umount lustre-MDT0001 complete [ 3287.872002] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 3288.817858] Key type lgssc unregistered [ 3288.926435] LNet: 11402:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3289.935713] LNet: Removed LNI 192.168.204.130@tcp [ 3299.719323] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 3304.409145] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3304.576857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3304.583862] alg: No test for adler32 (adler32-zlib) [ 3305.399727] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 3305.450548] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 3305.453464] LNet: Accept secure, port 988 [ 3306.989083] Key type lgssc registered [ 3307.253424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3309.613102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3311.398161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3313.031417] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3313.036427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3314.727448] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3314.737200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3316.759940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3316.782756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3316.788775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3317.868600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3317.878840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3317.913642] Lustre: lustre-MDT0000: new disk, initializing [ 3317.942998] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3317.950593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3318.829471] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3322.610937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3322.643391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3322.675322] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3322.686531] Lustre: 17147:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3322.701822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3322.703618] Lustre: Skipped 1 previous similar message [ 3322.735615] Lustre: lustre-MDT0001: new disk, initializing [ 3322.782845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3322.797012] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3322.802694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3323.764719] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3326.556011] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3327.545064] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3329.526890] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3329.533078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3329.555030] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3329.559467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3329.630309] Lustre: 18342:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3329.642442] Lustre: lustre-OST0000: new disk, initializing [ 3329.644519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3329.672460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3331.034955] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3333.645345] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3334.482463] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3335.028140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3335.031862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3335.051149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3335.566308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3336.477558] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3336.531009] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3340.045853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.046968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3340.047528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3340.060719] Lustre: Skipped 1 previous similar message [ 3343.230350] Lustre: server umount lustre-OST0000 complete [ 3345.052659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.061042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3345.063769] Lustre: Skipped 2 previous similar messages [ 3350.060997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3350.063399] Lustre: Skipped 1 previous similar message [ 3350.441333] Lustre: server umount lustre-MDT0000 complete [ 3351.614838] LustreError: 20025:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734258297 with bad export cookie 12793777286958078502 [ 3351.616585] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3351.622537] LustreError: 20025:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3351.760992] Lustre: server umount lustre-MDT0001 complete [ 3354.171272] Lustre: DEBUG MARKER: == conf-sanity-1 test 66: replace nids =================== 05:24:59 (1734258299) [ 3355.972721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3356.127101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3356.979491] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3359.203186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3359.244732] Lustre: 22037:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3360.132764] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3361.492967] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3362.329790] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3364.084203] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3364.088337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3364.151495] Lustre: 23147:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3364.182913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3364.185415] Lustre: Skipped 1 previous similar message [ 3365.472608] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3366.842626] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3373.981016] Lustre: Permanently deactivating lustre-OST0000 [ 3373.988015] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 3375.220902] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3375.800719] LustreError: 24283:0:(mgs_llog.c:1661:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 3375.804756] LustreError: 24283:0:(mgs_llog.c:1715:mgs_replace_nids()) Only MGS is allowed to be started [ 3375.807812] LustreError: 24283:0:(mgs_handler.c:941:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 3382.859131] Lustre: server umount lustre-OST0000 complete [ 3383.613448] LustreError: 24683:0:(mgs_llog.c:1715:mgs_replace_nids()) Only MGS is allowed to be started [ 3383.616402] LustreError: 24683:0:(mgs_llog.c:1715:mgs_replace_nids()) Skipped 1 previous similar message [ 3383.619571] LustreError: 24683:0:(mgs_handler.c:941:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 3383.622589] LustreError: 24683:0:(mgs_handler.c:941:mgs_iocontrol()) Skipped 1 previous similar message [ 3384.204515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.204945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3384.211547] Lustre: Skipped 2 previous similar messages [ 3389.212419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3389.214387] Lustre: Skipped 2 previous similar messages [ 3394.221307] LustreError: 21170:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3394.232106] LustreError: 21170:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 3394.237321] Lustre: server umount lustre-MDT0000 complete [ 3395.238695] LustreError: 21157:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734258341 with bad export cookie 12793777286958079370 [ 3395.240565] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3395.249893] LustreError: 21157:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3405.301424] Lustre: server umount lustre-MDT0001 complete [ 3407.511209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3408.308299] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3408.999190] LustreError: 26140:0:(mgs_handler.c:941:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 3410.696178] Lustre: 26600:0:(mgs_llog.c:1422:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.130@tcp [ 3411.274297] Lustre: server umount MGS complete [ 3414.484495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.643054] LustreError: 27554:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3414.648373] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3414.648375] Lustre: Skipped 1 previous similar message [ 3414.661886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3415.468996] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3417.771273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3417.805190] Lustre: 28416:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3417.866291] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3418.702850] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3419.968220] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3420.808515] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3421.370136] Lustre: Permanently reactivating lustre-OST0000 [ 3421.376301] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 3421.379081] Lustre: Skipped 2 previous similar messages [ 3444.137924] LustreError: 29407:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 3445.996255] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3446.000285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3446.058566] Lustre: 30909:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3446.092762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3446.094914] Lustre: Skipped 1 previous similar message [ 3447.362171] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3448.688891] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3469.117661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3469.117846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.118144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3469.118145] Lustre: Skipped 1 previous similar message [ 3469.131628] LustreError: Skipped 1 previous similar message [ 3470.710134] Lustre: server umount lustre-OST0000 complete [ 3472.957018] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3474.124870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.132840] Lustre: Skipped 2 previous similar messages [ 3477.883358] Lustre: server umount lustre-MDT0000 complete [ 3478.965545] LustreError: 27536:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734258425 with bad export cookie 12793777286958080273 [ 3478.966985] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3478.974606] LustreError: 27536:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3479.074128] Lustre: server umount lustre-MDT0001 complete [ 3482.365201] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 3482.987261] Key type lgssc unregistered [ 3483.083306] LNet: 326:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3484.089315] LNet: Removed LNI 192.168.204.130@tcp [ 3488.849907] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 3493.398788] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3493.519966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3493.551396] alg: No test for adler32 (adler32-zlib) [ 3494.373385] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 3494.432498] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 3494.436256] LNet: Accept secure, port 988 [ 3495.973050] Key type lgssc registered [ 3496.267515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3498.949770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3501.141857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3503.159628] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3503.167135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3505.222020] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3505.230290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3507.435108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3507.450718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3507.457267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3508.545775] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3508.560340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3508.593971] Lustre: lustre-MDT0000: new disk, initializing [ 3508.627007] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.636995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3509.675181] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3513.619888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3513.642385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3513.683706] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3513.693757] Lustre: 5042:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3513.711205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3513.713220] Lustre: Skipped 1 previous similar message [ 3513.748378] Lustre: lustre-MDT0001: new disk, initializing [ 3513.780921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3513.793868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3513.797402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3514.815190] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3517.601241] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3518.496263] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3520.581834] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3520.585971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3520.609835] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3520.613593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3520.686143] Lustre: 6238:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3520.699616] Lustre: lustre-OST0000: new disk, initializing [ 3520.702080] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3520.725619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3522.133475] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3524.007463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3524.011391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3524.036032] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3524.858740] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3525.751856] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3525.810231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3526.663287] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3526.715633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3529.036884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3529.036966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.037293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3529.046215] LustreError: Skipped 1 previous similar message [ 3533.309932] Lustre: server umount lustre-OST0000 complete [ 3538.813095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3538.816754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.821622] Lustre: Skipped 1 previous similar message [ 3538.824392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3538.828694] Lustre: Skipped 1 previous similar message [ 3540.536851] Lustre: server umount lustre-MDT0000 complete [ 3541.640711] LustreError: 7911:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734258487 with bad export cookie 5451770502665404939 [ 3541.641959] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3541.648827] LustreError: 7911:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3541.760522] Lustre: server umount lustre-MDT0001 complete [ 3543.960956] Lustre: DEBUG MARKER: == conf-sanity-1 test 70e: Sync-on-Cancel will be enabled by default on DNE ========================================================== 05:28:09 (1734258489) [ 3548.483306] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 3550.331809] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3552.849015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3554.735975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3556.319391] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3556.325987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3558.050234] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3558.056367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3559.783232] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3562.111534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3564.198752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3564.228869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3564.309709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3564.321475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3564.360044] Lustre: lustre-MDT0000: new disk, initializing [ 3564.390701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3564.397212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3565.331243] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3568.665148] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3568.670082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3570.586077] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3570.590432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3570.616226] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3570.624401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3570.699078] Lustre: 13821:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3570.712578] Lustre: lustre-OST0000: new disk, initializing [ 3570.714876] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3570.718134] Lustre: Skipped 1 previous similar message [ 3570.741104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3572.200748] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3573.000406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3573.004858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3573.015769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3575.041335] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3578.008584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3580.313614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3580.341359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3580.379700] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3580.393737] Lustre: 15364:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3580.409591] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3580.453290] Lustre: lustre-MDT0001: new disk, initializing [ 3580.498265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3580.511565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3581.563979] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3584.573457] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3584.630745] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3585.559521] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3586.639466] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 3588.807952] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.30@tcp (stopping) [ 3588.811095] Lustre: Skipped 2 previous similar messages [ 3591.166468] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3591.171267] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3591.178430] Lustre: Skipped 1 previous similar message [ 3591.181749] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3591.185914] Lustre: Skipped 1 previous similar message [ 3594.182082] Lustre: server umount lustre-MDT0001 complete [ 3601.180782] LustreError: 13849:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3601.187984] LustreError: 13849:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 3606.190421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3606.190438] LustreError: 12612:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3606.190439] LustreError: 12612:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 3606.210701] Lustre: Skipped 1 previous similar message [ 3606.214326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3606.217202] Lustre: Skipped 1 previous similar message [ 3611.197113] LustreError: 12612:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3611.958690] Lustre: server umount lustre-OST0000 complete [ 3619.307952] Lustre: server umount lustre-MDT0000 complete [ 3623.197666] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 3623.825207] Key type lgssc unregistered [ 3623.921304] LNet: 18127:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3624.927244] LNet: Removed LNI 192.168.204.130@tcp [ 3631.142111] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3631.266199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3631.291894] alg: No test for adler32 (adler32-zlib) [ 3632.107397] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 3632.165864] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 3632.168335] LNet: Accept secure, port 988 [ 3633.702001] Key type lgssc registered [ 3633.935328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3635.810432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3637.078405] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3637.082902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3638.644221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3639.953253] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3639.957750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3644.278899] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 3646.007561] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3648.505333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3650.536295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3652.362414] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3652.367422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3654.167775] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3654.174241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3656.150399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3656.173894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3656.181613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3657.266679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3657.277776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3657.312347] Lustre: lustre-MDT0000: new disk, initializing [ 3657.337911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3657.346333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3658.280723] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3662.343656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3662.369707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3662.400810] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3662.410997] Lustre: 24535:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3662.424535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3662.426904] Lustre: Skipped 1 previous similar message [ 3662.465034] Lustre: lustre-MDT0001: new disk, initializing [ 3662.495849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3662.509447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3662.515445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3663.451478] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3666.396821] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3667.361085] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3669.540606] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3669.554060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3669.580122] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3669.585446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3669.683556] Lustre: 25730:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3669.698694] Lustre: lustre-OST0000: new disk, initializing [ 3669.702304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3669.731875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3670.828940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3670.834764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3670.884853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3671.374829] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3674.224089] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3675.179343] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3675.249766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3676.225788] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3676.295076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3680.860623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3680.863588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3680.868974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3683.010873] Lustre: server umount lustre-OST0000 complete [ 3685.884874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3685.885502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3685.885504] Lustre: Skipped 1 previous similar message [ 3685.893923] Lustre: Skipped 2 previous similar messages [ 3690.190428] Lustre: server umount lustre-MDT0000 complete [ 3691.255871] LustreError: 24549:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734258637 with bad export cookie 18109093529116511453 [ 3691.257229] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3691.267401] LustreError: 24549:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3691.369573] Lustre: server umount lustre-MDT0001 complete [ 3693.655054] Lustre: DEBUG MARKER: == conf-sanity-1 test 78: run resize2fs on MDT and OST filesystems ========================================================== 05:30:39 (1734258639) [ 3696.400314] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 3698.015510] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3700.683883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3702.373257] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3702.379005] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3704.367786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3704.389160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.463604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3704.475012] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3704.509924] Lustre: lustre-MDT0000: new disk, initializing [ 3704.541303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.549524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3705.423464] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3708.117240] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3710.049949] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3710.054716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3710.075867] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3710.080737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3710.155156] Lustre: 31363:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3710.168714] Lustre: lustre-OST0000: new disk, initializing [ 3710.172569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3710.175032] Lustre: Skipped 1 previous similar message [ 3710.197446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3710.434810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3710.438497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3710.451012] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3711.588044] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3714.330835] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3720.881853] Lustre: DEBUG MARKER: create test files [ 3730.460689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.466181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3730.466503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3730.466504] Lustre: Skipped 1 previous similar message [ 3733.174874] Lustre: server umount lustre-OST0000 complete [ 3734.351687] Lustre: server umount lustre-MDT0000 complete [ 3737.607690] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 3738.257329] Key type lgssc unregistered [ 3738.358685] LNet: 667:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3739.364269] LNet: Removed LNI 192.168.204.130@tcp [ 3748.087849] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3748.249713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3748.254931] alg: No test for adler32 (adler32-zlib) [ 3749.079570] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 3749.140982] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 3749.144134] LNet: Accept secure, port 988 [ 3750.687339] Key type lgssc registered [ 3750.963998] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3753.138046] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3753.147348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3754.318610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3755.295720] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3756.863513] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3759.266164] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3759.273537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3759.363474] Lustre: 3726:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3759.417936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3761.102256] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3762.626123] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3764.421161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:88 to 0x240000400:129) [ 3764.452495] Lustre: 3734:0:(osd_internal.h:1314:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 3764.457577] Lustre: 3734:0:(osd_handler.c:1955:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 3764.461930] Lustre: 3734:0:(osd_handler.c:1962:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 3764.465334] Lustre: 3734:0:(osd_handler.c:1972:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 3764.468967] Lustre: 3734:0:(osd_handler.c:1979:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 3764.473529] Lustre: 3734:0:(osd_handler.c:1986:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3770.499370] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 3771.820234] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 3774.428422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3774.428483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3774.428889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3779.436694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3779.513852] Lustre: server umount lustre-OST0000 complete [ 3780.777942] Lustre: server umount lustre-MDT0000 complete [ 3784.094433] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 3784.770400] Key type lgssc unregistered [ 3784.863953] LNet: 5475:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3785.870251] LNet: Removed LNI 192.168.204.130@tcp [ 3794.755884] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3794.907934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3794.928858] alg: No test for adler32 (adler32-zlib) [ 3795.748140] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 3795.801471] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 3795.804404] LNet: Accept secure, port 988 [ 3797.341024] Key type lgssc registered [ 3797.562684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3799.560344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3799.567124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3800.707867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.543001] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3803.006961] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3804.927329] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3804.931402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3804.999345] Lustre: 8642:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3805.040746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.358659] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3807.774353] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3811.037240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 3811.065477] Lustre: 8649:0:(osd_internal.h:1314:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 3811.069374] Lustre: 8649:0:(osd_handler.c:1955:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 3811.072658] Lustre: 8649:0:(osd_handler.c:1962:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 3811.076631] Lustre: 8649:0:(osd_handler.c:1972:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 3811.079978] Lustre: 8649:0:(osd_handler.c:1979:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 3811.083359] Lustre: 8649:0:(osd_handler.c:1986:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3815.537716] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 3821.052451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3821.052452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3821.052765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3823.203188] Lustre: server umount lustre-OST0000 complete [ 3824.341448] Lustre: server umount lustre-MDT0000 complete [ 3827.640289] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 3828.410769] Key type lgssc unregistered [ 3828.515916] LNet: 10317:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3829.524090] LNet: Removed LNI 192.168.204.130@tcp [ 3834.355122] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 3838.792930] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3838.914935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3838.941064] alg: No test for adler32 (adler32-zlib) [ 3839.756200] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 3839.807994] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 3839.810001] LNet: Accept secure, port 988 [ 3841.338958] Key type lgssc registered [ 3841.589323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3844.054004] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3845.947601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3847.703153] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3847.707816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3849.608437] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3849.613509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3851.670854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3851.693385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3851.699240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3852.782911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3852.796550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3852.835773] Lustre: lustre-MDT0000: new disk, initializing [ 3852.866632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3852.874729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3853.841560] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3857.532038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3857.563062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3857.595657] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3857.606330] Lustre: 14983:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3857.619517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3857.621821] Lustre: Skipped 1 previous similar message [ 3857.655673] Lustre: lustre-MDT0001: new disk, initializing [ 3857.682492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3857.696088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3857.701064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3858.606895] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3861.309583] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3862.266977] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3864.313422] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3864.319207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3864.347371] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3864.352905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3864.430006] Lustre: 16178:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3864.445478] Lustre: lustre-OST0000: new disk, initializing [ 3864.447744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3864.471495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3865.912103] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3868.853489] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3869.824847] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3870.478413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3870.482546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3870.501534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3870.903739] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3871.821772] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3871.878058] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3875.500993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3875.501075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3875.501774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3875.513062] LustreError: Skipped 1 previous similar message [ 3878.519269] Lustre: server umount lustre-OST0000 complete [ 3880.508711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3880.513960] Lustre: Skipped 1 previous similar message [ 3880.514703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3880.514704] Lustre: Skipped 1 previous similar message [ 3885.516410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3885.518898] Lustre: Skipped 2 previous similar messages [ 3885.608561] Lustre: server umount lustre-MDT0000 complete [ 3886.740864] LustreError: 14997:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734258833 with bad export cookie 313808999209537059 [ 3886.742598] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3886.751585] LustreError: 14997:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3886.860935] Lustre: server umount lustre-MDT0001 complete [ 3889.288706] Lustre: DEBUG MARKER: == conf-sanity-1 test 79: format MDT/OST without mgs option (should return errors) ========================================================== 05:33:54 (1734258834) [ 3891.511979] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3895.488131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3902.929941] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 3904.781448] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3907.332111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3909.249593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3910.945690] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3910.952164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3912.688404] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3912.692602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3914.838599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3914.868898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.960039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3914.979268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3915.019475] Lustre: lustre-MDT0000: new disk, initializing [ 3915.043425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3915.050083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3916.012355] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3919.788953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3919.821013] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3919.850980] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3919.859148] Lustre: 24286:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3919.872116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3919.874651] Lustre: Skipped 1 previous similar message [ 3919.907861] Lustre: lustre-MDT0001: new disk, initializing [ 3919.936260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3919.946843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3919.951450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3920.862426] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3923.557987] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3924.408934] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3926.269282] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3926.273434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3926.297756] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3926.301395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3926.377553] Lustre: 25484:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3927.809524] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3930.513121] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3931.379157] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3934.123397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3934.126760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3934.144679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3934.489311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3935.405382] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3935.460509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3939.150448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3939.150470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3939.150880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3939.160165] Lustre: Skipped 1 previous similar message [ 3942.074315] Lustre: server umount lustre-OST0000 complete [ 3944.156746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3944.157258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3944.157259] Lustre: Skipped 1 previous similar message [ 3944.164864] Lustre: Skipped 1 previous similar message [ 3949.198453] Lustre: server umount lustre-MDT0000 complete [ 3950.321628] LustreError: 23337:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734258896 with bad export cookie 313808999209537927 [ 3950.322812] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3950.330813] LustreError: 23337:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3950.432553] Lustre: server umount lustre-MDT0001 complete [ 3952.769878] Lustre: DEBUG MARKER: == conf-sanity-1 test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 05:34:58 (1734258898) [ 3953.203459] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 3954.582515] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3954.588690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3954.905480] loop: module loaded [ 3954.912211] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 3954.921117] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3955.315030] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 3955.883565] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 3956.839308] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 3959.269360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3960.374930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3961.382319] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3961.386594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3962.434608] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3962.440143] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3964.285191] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3964.289176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3964.306953] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3964.311654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3964.319463] LustreError: 30386:0:(obd_config.c:797:class_setup()) setup lustre-OST0000-osd failed (-28) [ 3964.322332] LustreError: 30386:0:(obd_mount.c:214:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 3964.324478] LustreError: 30386:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 3964.327576] LustreError: 30386:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 3969.382477] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 3971.096237] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3973.477387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3975.318579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3976.976868] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3976.981370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3978.598526] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3978.602018] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3980.410202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3980.427845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3980.494463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3980.502451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3980.504378] Lustre: Skipped 1 previous similar message [ 3980.534739] Lustre: lustre-MDT0000: new disk, initializing [ 3980.536356] Lustre: Skipped 1 previous similar message [ 3980.557864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3980.560320] Lustre: Skipped 1 previous similar message [ 3980.566397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3981.315883] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3984.769420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3984.787535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3984.816279] Lustre: 2423:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3984.885992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3985.658394] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3988.154368] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3988.945770] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3990.613865] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3990.617208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3990.636262] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3990.639400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3991.978172] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3994.559737] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3994.725283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3994.746816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3995.444143] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3995.498143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3996.338755] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3996.391906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3999.740565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3999.743367] LustreError: Skipped 1 previous similar message [ 3999.745028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.750390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3999.753939] Lustre: Skipped 3 previous similar messages [ 4002.975264] Lustre: server umount lustre-OST0000 complete [ 4010.076485] Lustre: server umount lustre-MDT0000 complete [ 4011.116663] LustreError: 1476:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734258957 with bad export cookie 313808999209538802 [ 4011.120148] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4011.125044] LustreError: 1476:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4011.228878] Lustre: server umount lustre-MDT0001 complete [ 4013.355135] Lustre: DEBUG MARKER: == conf-sanity-1 test 86: Replacing mkfs.lustre -G option ========================================================== 05:35:58 (1734258958) [ 4018.089295] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4018.095484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4022.510122] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 4024.072811] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4026.467753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4028.183858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4029.818339] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4029.824100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4031.482234] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4031.487046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4033.384074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4033.402913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4033.474512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4033.477404] Lustre: Skipped 1 previous similar message [ 4033.487969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4033.490119] Lustre: Skipped 3 previous similar messages [ 4033.524250] Lustre: lustre-MDT0000: new disk, initializing [ 4033.526032] Lustre: Skipped 2 previous similar messages [ 4033.549525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4033.552448] Lustre: Skipped 2 previous similar messages [ 4033.558340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4033.561573] Lustre: Skipped 2 previous similar messages [ 4034.439476] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4038.180959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4038.209014] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4038.243178] Lustre: 11449:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4038.247990] Lustre: 11449:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4038.331508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4039.201172] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4041.939142] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4042.893893] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4044.821986] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4044.831390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4044.858567] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4044.864262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4046.411653] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4047.342980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4049.117293] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4050.007499] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4050.063283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4050.896782] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4050.953926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4052.332718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4052.336040] LustreError: Skipped 1 previous similar message [ 4052.337706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4052.343256] Lustre: Skipped 3 previous similar messages [ 4052.346073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4052.348630] Lustre: Skipped 5 previous similar messages [ 4057.588747] Lustre: server umount lustre-OST0000 complete [ 4064.680318] Lustre: server umount lustre-MDT0000 complete [ 4065.720724] LustreError: 14334:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734259012 with bad export cookie 313808999209539670 [ 4065.722099] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4065.729503] LustreError: 14334:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4065.827111] Lustre: server umount lustre-MDT0001 complete [ 4067.990249] Lustre: DEBUG MARKER: == conf-sanity-1 test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 05:36:53 (1734259013) [ 4069.643280] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 4070.288865] Key type lgssc unregistered [ 4070.376280] LNet: 15398:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4071.384017] LNet: Removed LNI 192.168.204.130@tcp [ 4076.262218] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 4080.648084] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4080.833271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4080.845771] alg: No test for adler32 (adler32-zlib) [ 4081.665301] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 4081.717634] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 4081.719987] LNet: Accept secure, port 988 [ 4083.252084] Key type lgssc registered [ 4083.473671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4085.738005] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4087.589076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4089.261860] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4089.266690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4090.971738] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4090.979537] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4092.505281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4094.411770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4094.437072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4094.445282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4095.525014] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4095.537398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4095.574663] Lustre: lustre-MDT0000: new disk, initializing [ 4095.600886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4095.608472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4096.444991] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4100.188124] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4100.193475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4100.217076] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4100.221242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4100.302074] Lustre: 20345:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4100.315182] Lustre: lustre-OST0000: new disk, initializing [ 4100.317150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4100.319540] Lustre: Skipped 1 previous similar message [ 4100.340245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4101.755040] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4101.835248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4101.839078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4101.854968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4105.371917] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4105.376319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4105.390586] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4105.394659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4105.430222] Lustre: 21415:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4105.443248] Lustre: lustre-OST0001: new disk, initializing [ 4105.445574] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4105.467219] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4106.867629] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4110.192099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 4110.195832] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 4110.205942] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 4110.779140] Lustre: server umount lustre-MDT0000 complete [ 4112.747299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4114.125221] LustreError: 19397:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734259060 with bad export cookie 255942179595663762 [ 4114.130272] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4124.151560] Lustre: server umount lustre-OST0000 complete [ 4135.306265] Lustre: server umount lustre-OST0001 complete [ 4137.470479] Lustre: DEBUG MARKER: == conf-sanity-1 test 89: check tunefs --param and --erase-param{s} options ========================================================== 05:38:02 (1734259082) [ 4140.839542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4141.402736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4141.976776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4142.550267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4142.843807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4143.411472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4148.011180] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 4149.828922] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4152.236095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4154.052347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4155.639321] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4155.643180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4157.269753] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4157.274454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4159.141597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4159.167019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4159.238868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4159.248106] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4159.282560] Lustre: lustre-MDT0000: new disk, initializing [ 4159.303216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4159.309620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4160.144196] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4163.652432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4163.676387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4163.703973] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4163.711813] Lustre: 29414:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4163.725043] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4163.727284] Lustre: Skipped 1 previous similar message [ 4163.760440] Lustre: lustre-MDT0001: new disk, initializing [ 4163.786542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4163.798544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4163.802042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4164.622193] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4167.216360] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4168.095148] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4169.937081] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4169.941435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4169.958201] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4169.961871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4170.032078] Lustre: 30610:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4171.382808] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4172.424045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4172.428353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4172.454374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4173.985915] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4174.830298] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4174.882749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4175.695506] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4175.746217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4177.452610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4177.452863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.453150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4177.461861] LustreError: Skipped 1 previous similar message [ 4182.323713] Lustre: server umount lustre-OST0000 complete [ 4183.804431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4183.807261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4183.811820] Lustre: Skipped 1 previous similar message [ 4183.814296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4183.817040] Lustre: Skipped 1 previous similar message [ 4187.468919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4187.469674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.475719] Lustre: Skipped 1 previous similar message [ 4189.366525] Lustre: server umount lustre-MDT0000 complete [ 4190.380615] LustreError: 29428:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734259136 with bad export cookie 255942179595664966 [ 4190.381602] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4190.387999] LustreError: 29428:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4190.493695] Lustre: server umount lustre-MDT0001 complete [ 4192.562775] Lustre: DEBUG MARKER: == conf-sanity-1 test 93: register mulitple MDT at the same time ========================================================== 05:38:58 (1734259138) [ 4192.920435] Lustre: DEBUG MARKER: SKIP: conf-sanity-1 test_93 needs >= 3 MDTs [ 4193.320141] Lustre: DEBUG MARKER: == conf-sanity-1 test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 05:38:58 (1734259138) [ 4197.251687] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 4198.892055] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4201.160096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4202.839910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4204.340443] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4204.346704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4205.841480] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4205.845967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4207.537316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4207.553986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.617689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4207.626095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4207.628546] Lustre: Skipped 1 previous similar message [ 4207.660121] Lustre: lustre-MDT0000: new disk, initializing [ 4207.661501] Lustre: Skipped 1 previous similar message [ 4207.677425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4207.679525] Lustre: Skipped 1 previous similar message [ 4207.683496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4208.437230] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4211.944159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4211.963302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4211.989317] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4211.996558] Lustre: 4920:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4212.069803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4212.880630] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4215.477283] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4216.318876] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4218.035241] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4218.038588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4218.056743] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4218.060415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4219.389448] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4221.855047] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4223.623987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4223.640080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4228.636461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4228.639687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.644227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4230.527375] Lustre: server umount lustre-OST0000 complete [ 4232.092510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4232.097335] LustreError: Skipped 1 previous similar message [ 4233.582695] Lustre: server umount lustre-MDT0000 complete [ 4233.660402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4233.665061] Lustre: Skipped 2 previous similar messages [ 4234.588605] LustreError: 4935:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734259180 with bad export cookie 255942179595665834 [ 4234.589975] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4234.596874] LustreError: 4935:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4234.693542] Lustre: server umount lustre-MDT0001 complete [ 4237.742045] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 4238.324178] Key type lgssc unregistered [ 4238.411109] LNet: 8263:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4239.419036] LNet: Removed LNI 192.168.204.130@tcp [ 4245.405871] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 4249.530397] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4249.720286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4249.730061] alg: No test for adler32 (adler32-zlib) [ 4250.535879] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 4250.588018] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 4250.589465] LNet: Accept secure, port 988 [ 4252.119978] Key type lgssc registered [ 4252.314480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4254.256678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4255.696008] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4257.031029] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4257.034503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4258.389048] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4258.392184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4260.029054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4260.042580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4260.044723] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 4260.044723] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 4260.044723] [ 4260.048893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 4261.109296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4261.117140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4261.147764] Lustre: lustre-MDT0000: new disk, initializing [ 4261.164476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4261.169638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4261.920317] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4265.281500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4265.298254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4265.325909] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4265.332603] Lustre: 13255:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4265.343765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4265.346571] Lustre: Skipped 1 previous similar message [ 4265.379344] Lustre: lustre-MDT0001: new disk, initializing [ 4265.402175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4265.410947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4265.413924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4266.189153] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4269.513176] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4269.516152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4269.534486] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4269.537881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4269.596475] Lustre: 14186:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4269.604910] Lustre: lustre-OST0000: new disk, initializing [ 4269.606710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4269.621172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4270.073250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4270.075878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4270.094287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4270.847898] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4274.050631] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4274.052823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4274.065556] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4274.068046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4274.096259] Lustre: 15262:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4274.105196] Lustre: lustre-OST0001: new disk, initializing [ 4274.107113] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4274.124892] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4275.368524] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4278.465484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4278.468637] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4278.486205] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4279.721914] Lustre: DEBUG MARKER: == conf-sanity-1 test 105: check file creation for ro and rw bind mnt pt ========================================================== 05:40:25 (1734259225) [ 4293.501070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.501584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4293.507600] Lustre: Skipped 1 previous similar message [ 4296.430634] Lustre: server umount lustre-OST0000 complete [ 4298.508581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4298.509088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4298.509090] Lustre: Skipped 1 previous similar message [ 4298.516409] Lustre: Skipped 1 previous similar message [ 4303.507427] Lustre: server umount lustre-MDT0000 complete [ 4303.516439] LustreError: 16154:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4303.521566] LustreError: 16154:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 5 previous similar messages [ 4304.570627] LustreError: 12310:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734259250 with bad export cookie 8474709106139043766 [ 4304.572453] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4304.579509] LustreError: 12310:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4308.525222] LustreError: 16154:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.525994] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4308.525996] Lustre: Skipped 1 previous similar message [ 4308.526454] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4308.526456] Lustre: Skipped 2 previous similar messages [ 4309.528315] LustreError: 16154:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4310.686491] Lustre: server umount lustre-MDT0001 complete [ 4313.881935] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 4322.842893] Lustre: server umount lustre-OST0001 complete [ 4324.213161] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 4328.434571] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4330.782524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4332.519221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4334.149899] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4334.153761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4335.717131] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4335.719704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4337.355731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4337.374528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4337.436759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4337.444349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4337.476356] Lustre: lustre-MDT0000: new disk, initializing [ 4337.494954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4337.499920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4338.246022] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4341.520983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4341.537597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4341.559311] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4341.565482] Lustre: 21492:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4341.628861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4342.364057] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4344.798190] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4345.550939] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4347.163887] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4347.167862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4347.185260] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4347.188904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4347.252861] Lustre: 22689:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4347.263296] Lustre: lustre-OST0000: new disk, initializing [ 4347.264800] Lustre: Skipped 1 previous similar message [ 4347.266627] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4347.268274] Lustre: Skipped 2 previous similar messages [ 4347.288764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4347.291886] Lustre: Skipped 1 previous similar message [ 4348.486528] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4350.901598] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4351.800885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4351.804742] Lustre: Skipped 1 previous similar message [ 4351.806363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4351.821373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4361.836511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4361.836750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4361.836952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4361.836953] Lustre: Skipped 1 previous similar message [ 4361.846894] Lustre: Skipped 1 previous similar message [ 4363.754517] Lustre: server umount lustre-OST0000 complete [ 4366.652546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4366.656151] LustreError: Skipped 1 previous similar message [ 4366.658249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.663557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4366.666931] Lustre: Skipped 1 previous similar message [ 4370.879218] Lustre: server umount lustre-MDT0000 complete [ 4371.852338] LustreError: 20563:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4371.858531] LustreError: 20563:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 4371.899760] LustreError: 21506:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734259318 with bad export cookie 8474709106139044963 [ 4371.901117] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4371.908996] LustreError: 21506:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4372.017506] Lustre: server umount lustre-MDT0001 complete [ 4375.079213] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 4375.687529] Key type lgssc unregistered [ 4375.780193] LNet: 24649:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4376.786188] LNet: Removed LNI 192.168.204.130@tcp [ 4382.839045] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 4387.040002] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4387.159290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4387.184722] alg: No test for adler32 (adler32-zlib) [ 4388.006698] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 4388.061561] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 4388.063615] LNet: Accept secure, port 988 [ 4389.595961] Key type lgssc registered [ 4389.822194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4392.044125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4393.765593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4395.370640] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4395.375028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4396.930969] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4396.936600] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4398.659276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4398.672098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4398.676497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.746168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4399.755272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4399.788003] Lustre: lustre-MDT0000: new disk, initializing [ 4399.809134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.816828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4400.624625] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4404.117549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4404.134075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4404.161382] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4404.170177] Lustre: 29576:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4404.182607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4404.184434] Lustre: Skipped 1 previous similar message [ 4404.216497] Lustre: lustre-MDT0001: new disk, initializing [ 4404.241881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4404.251699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4404.255026] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4405.038960] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4407.591750] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4408.408003] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4410.175222] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4410.179454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4410.196211] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4410.199916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4410.268294] Lustre: 30772:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4410.279488] Lustre: lustre-OST0000: new disk, initializing [ 4410.281608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4410.304393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4411.594157] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4414.064512] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4415.744369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4415.747683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4415.764300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4487.611848] LustreError: 30782:0:(ofd_dev.c:1686:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 4487.616573] LustreError: 31536:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 4487.643165] LustreError: 28641:0:(lod_qos.c:1380:lod_ost_alloc_specific()) can't lstripe objid [0x200000402:0x7f33:0x0]: have 0 want 1 [ 4608.661227] LustreError: 32052:0:(ofd_dev.c:1686:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 4608.663950] LustreError: 31536:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 4608.684308] LustreError: 29596:0:(lod_qos.c:1380:lod_ost_alloc_specific()) can't lstripe objid [0x200000402:0xfe64:0x0]: have 0 want 1 [ 4728.633975] LustreError: 32053:0:(ofd_dev.c:1686:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 4728.638001] LustreError: 31536:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 4728.674987] LustreError: 28643:0:(lod_qos.c:1380:lod_ost_alloc_specific()) can't lstripe objid [0x200000402:0x17d95:0x0]: have 0 want 1 [ 4853.801836] LustreError: 30782:0:(ofd_dev.c:1686:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 4853.804649] LustreError: 31536:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 4853.832396] LustreError: 28641:0:(lod_qos.c:1380:lod_ost_alloc_specific()) can't lstripe objid [0x200000402:0x1fcc6:0x0]: have 0 want 1 [ 4972.773310] LustreError: 32053:0:(ofd_dev.c:1686:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 4972.777142] LustreError: 31536:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 4972.803495] LustreError: 28642:0:(lod_qos.c:1380:lod_ost_alloc_specific()) can't lstripe objid [0x200000403:0x7bf7:0x0]: have 0 want 1 [ 5061.756477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5061.758519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5061.762165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5065.261476] Lustre: server umount lustre-OST0000 complete [ 5066.796447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5066.796867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5066.796869] Lustre: Skipped 1 previous similar message [ 5066.802157] Lustre: Skipped 2 previous similar messages [ 5071.804279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5071.807621] Lustre: Skipped 2 previous similar messages [ 5072.287465] Lustre: server umount lustre-MDT0000 complete [ 5073.256473] LustreError: 28630:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734260019 with bad export cookie 1439193169507800702 [ 5073.257345] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5073.265645] LustreError: 28630:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5073.362012] Lustre: server umount lustre-MDT0001 complete [ 5076.292740] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 5076.978943] Key type lgssc unregistered [ 5077.069121] LNet: 6789:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5078.075066] LNet: Removed LNI 192.168.204.130@tcp [ 5086.458169] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 5090.671482] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5090.835435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5090.839716] alg: No test for adler32 (adler32-zlib) [ 5091.642748] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 5091.689434] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 5091.690964] LNet: Accept secure, port 988 [ 5093.214952] Key type lgssc registered [ 5093.416284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5095.723509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5097.386723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5098.866108] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5098.869801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5100.736036] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5100.741745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5102.639876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5102.654073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5102.659802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5103.734864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5103.743137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5103.775775] Lustre: lustre-MDT0000: new disk, initializing [ 5103.791674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5103.798135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5104.604377] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5108.325772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5108.347500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5108.372232] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5108.378128] Lustre: 12515:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5108.389312] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5108.392036] Lustre: Skipped 1 previous similar message [ 5108.429104] Lustre: lustre-MDT0001: new disk, initializing [ 5108.452219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5108.461867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5108.465190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5109.179625] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5111.770478] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5112.612665] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5114.428126] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5114.431816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5114.451051] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5114.456346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5114.558421] Lustre: 13711:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5114.577203] Lustre: lustre-OST0000: new disk, initializing [ 5114.581463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5114.612468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5114.975134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5114.981724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5115.006170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5115.998757] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5118.551609] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5126.640361] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 5134.958604] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 5140.362428] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 5180.092919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5180.093912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5180.094399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5185.101419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5185.104304] Lustre: Skipped 2 previous similar messages [ 5185.986781] Lustre: server umount lustre-OST0000 complete [ 5188.572528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5188.575048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5188.579493] Lustre: Skipped 1 previous similar message [ 5188.581719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5190.108571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5193.038687] Lustre: server umount lustre-MDT0000 complete [ 5194.038896] LustreError: 11569:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734260140 with bad export cookie 13521424144549870365 [ 5194.040018] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5194.049269] LustreError: 11569:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5194.172258] Lustre: server umount lustre-MDT0001 complete [ 5196.414037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5197.193720] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5198.454504] Lustre: server umount MGS complete [ 5201.217058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5201.342344] LustreError: 18846:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 5201.357750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5202.092400] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5204.276694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5204.300176] Lustre: 19696:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5205.118544] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5206.353367] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5207.202547] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5208.880195] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5208.882989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5208.943082] Lustre: 20804:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5208.975561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5208.977526] Lustre: Skipped 1 previous similar message [ 5210.171084] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5211.422621] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5231.004443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5231.006876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.011769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5231.014509] Lustre: Skipped 2 previous similar messages [ 5234.012650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5236.182087] Lustre: server umount lustre-OST0000 complete [ 5239.020459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5239.020865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5239.020867] Lustre: Skipped 2 previous similar messages [ 5239.026740] Lustre: Skipped 2 previous similar messages [ 5243.172742] Lustre: server umount lustre-MDT0000 complete [ 5244.028230] LustreError: 18851:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5244.035073] LustreError: 18851:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 5244.163640] LustreError: 18815:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734260190 with bad export cookie 13521424144549872199 [ 5244.164818] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5244.170030] LustreError: 18815:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5244.275090] Lustre: server umount lustre-MDT0001 complete [ 5247.230018] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 5247.875075] Key type lgssc unregistered [ 5247.953155] LNet: 23225:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5248.958088] LNet: Removed LNI 192.168.204.130@tcp [ 5257.469250] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 5261.664469] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5261.805136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5261.808707] alg: No test for adler32 (adler32-zlib) [ 5262.612778] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 5262.661186] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 5262.663190] LNet: Accept secure, port 988 [ 5264.192965] Key type lgssc registered [ 5264.392490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5266.392826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5267.999317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5269.443762] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5269.448916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5270.933343] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5270.936811] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5272.801377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5272.818339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5272.824040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5273.890493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5273.900390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5273.935669] Lustre: lustre-MDT0000: new disk, initializing [ 5273.953039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5273.958226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5274.721129] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5278.415235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5278.429470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5278.458668] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5278.466636] Lustre: 28942:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5278.478753] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5278.481480] Lustre: Skipped 1 previous similar message [ 5278.521212] Lustre: lustre-MDT0001: new disk, initializing [ 5278.545047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5278.554131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5278.556477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5279.393420] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5282.119780] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5282.914854] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5284.776015] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5284.781023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5284.803610] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5284.806484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5284.871559] Lustre: 30138:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5284.881144] Lustre: lustre-OST0000: new disk, initializing [ 5284.883988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5284.902420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5286.122342] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5287.179723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5287.184640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5287.203608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5288.642712] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5296.655962] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 5305.012601] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 5310.525782] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 5352.301099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5352.301153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5352.301897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5352.314966] LustreError: Skipped 1 previous similar message [ 5357.308984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5357.311838] Lustre: Skipped 2 previous similar messages [ 5358.350071] Lustre: server umount lustre-OST0000 complete [ 5362.316549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5362.316806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5362.322894] Lustre: Skipped 2 previous similar messages [ 5365.770904] Lustre: server umount lustre-MDT0000 complete [ 5366.785512] LustreError: 27996:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734260313 with bad export cookie 15518129932294559537 [ 5366.786537] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5366.791042] LustreError: 27996:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5366.890201] Lustre: server umount lustre-MDT0001 complete [ 5369.204685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5370.155495] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5371.607890] Lustre: server umount MGS complete [ 5375.040543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5375.166470] LustreError: 2849:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 5375.181203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5375.991883] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5378.374012] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5378.400133] Lustre: 3703:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5379.236826] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5380.547229] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5381.371187] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5383.182360] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5383.188297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5383.264722] Lustre: 4824:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5383.310284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5383.314424] Lustre: Skipped 1 previous similar message [ 5384.619483] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5385.829496] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5403.340377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5403.342312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5403.345579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5403.347558] Lustre: Skipped 2 previous similar messages [ 5407.931533] Lustre: server umount lustre-OST0000 complete [ 5413.356390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5413.359559] Lustre: Skipped 1 previous similar message [ 5413.360895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5413.362849] Lustre: Skipped 1 previous similar message [ 5414.956602] Lustre: server umount lustre-MDT0000 complete [ 5416.096834] LustreError: 2819:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734260362 with bad export cookie 15518129932294561364 [ 5416.098732] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5416.107543] LustreError: 2819:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5416.220962] Lustre: server umount lustre-MDT0001 complete [ 5419.568680] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 5420.154635] Key type lgssc unregistered [ 5420.254102] LNet: 7246:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5421.262056] LNet: Removed LNI 192.168.204.130@tcp [ 5431.408988] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5431.566295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5431.591160] alg: No test for adler32 (adler32-zlib) [ 5432.398509] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 5432.443950] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 5432.445678] LNet: Accept secure, port 988 [ 5433.971993] Key type lgssc registered [ 5434.272652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5436.276340] SGI XFS with ACLs, security attributes, no debug enabled [ 5436.281600] XFS (loop0): Mounting V5 Filesystem [ 5436.290183] XFS (loop0): Ending clean mount [ 5438.555274] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [ 5438.671444] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5442.720728] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 5444.410419] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5446.634605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5448.792229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5450.673532] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5450.679302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5452.731823] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5452.737156] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5454.796541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5454.814224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5454.820111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.898812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5455.907570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5455.944247] Lustre: lustre-MDT0000: new disk, initializing [ 5455.966239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5455.972820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5456.912247] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5460.656864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5460.677082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5460.697742] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5460.704637] Lustre: 13840:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5460.716820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5460.718696] Lustre: Skipped 1 previous similar message [ 5460.751571] Lustre: lustre-MDT0001: new disk, initializing [ 5460.773991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5460.784424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5460.789110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5461.554545] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5464.374984] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5465.107265] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5466.838925] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5466.841755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5466.859297] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5466.861904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5466.929670] Lustre: 15036:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5466.939108] Lustre: lustre-OST0000: new disk, initializing [ 5466.940596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5466.958393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5468.254733] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5471.223773] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5471.999690] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5472.662965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5472.669952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5472.691007] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5473.090416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5474.058017] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5474.107962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5477.692960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5477.693524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5477.694197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5477.709861] LustreError: Skipped 1 previous similar message [ 5480.659226] Lustre: server umount lustre-OST0000 complete [ 5482.700366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5482.700644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5482.700645] Lustre: Skipped 1 previous similar message [ 5482.705593] Lustre: Skipped 2 previous similar messages [ 5487.708405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5487.710641] Lustre: Skipped 2 previous similar messages [ 5487.823871] Lustre: server umount lustre-MDT0000 complete [ 5489.046371] LustreError: 12891:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734260435 with bad export cookie 13643171809901793575 [ 5489.048877] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5489.060431] LustreError: 12891:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5489.232699] Lustre: server umount lustre-MDT0001 complete [ 5491.438992] XFS (loop0): Unmounting Filesystem [ 5492.772590] Lustre: DEBUG MARKER: == conf-sanity-1 test 122a: Check OST sequence update ==== 06:00:38 (1734260438) [ 5497.104556] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 5498.576418] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5501.351767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5503.091671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5504.856576] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5504.863172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5506.379841] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5506.383328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5507.854973] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5510.290329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5510.311422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5510.375552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5510.382740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5510.419553] Lustre: lustre-MDT0000: new disk, initializing [ 5510.436396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5510.441088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5511.205740] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5515.002649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5515.024818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5515.067316] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5515.079396] Lustre: 22323:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5515.101562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5515.105580] Lustre: Skipped 1 previous similar message [ 5515.159057] Lustre: lustre-MDT0001: new disk, initializing [ 5515.188738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5515.200450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5515.208325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5516.032998] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5518.764122] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5519.574521] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5521.263243] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5521.268050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5521.288618] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5521.292558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5521.364063] Lustre: 23523:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5522.582717] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5525.038082] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5526.116816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5526.119676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5526.127562] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 5526.139053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5526.144381] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 5532.912651] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 5536.140301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5536.142468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5536.148014] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5541.164423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5541.166050] Lustre: Skipped 2 previous similar messages [ 5542.216040] Lustre: server umount lustre-OST0000 complete [ 5545.228686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5545.232253] LustreError: Skipped 1 previous similar message [ 5545.234048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.245047] Lustre: Skipped 1 previous similar message [ 5549.255144] Lustre: server umount lustre-MDT0000 complete [ 5550.221621] LustreError: 21377:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734260496 with bad export cookie 13643171809901794443 [ 5550.223202] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5550.232551] LustreError: 21377:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5550.362607] Lustre: server umount lustre-MDT0001 complete [ 5553.277241] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 5553.835159] Key type lgssc unregistered [ 5553.919069] LNet: 25740:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5554.924096] LNet: Removed LNI 192.168.204.130@tcp [ 5562.597001] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5562.711285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5562.740595] alg: No test for adler32 (adler32-zlib) [ 5563.575334] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 5563.637427] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 5563.639227] LNet: Accept secure, port 988 [ 5565.172033] Key type lgssc registered [ 5565.406909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5568.294308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5568.302281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.437721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5570.111287] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5572.873480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5572.905405] Lustre: 28462:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5572.978294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5573.762417] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5574.591936] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 5576.496220] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5576.499612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5576.566454] Lustre: 29381:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5576.612338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5577.830455] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5580.023852] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5580.027598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5580.050190] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5580.054287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5580.088147] Lustre: 30371:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5580.099157] Lustre: lustre-OST0001: new disk, initializing [ 5580.101061] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5580.122983] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5581.358004] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5583.123754] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 5584.655896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5584.658404] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5584.672438] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5586.182621] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5593.652869] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 5603.164444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5603.167262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.171954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5604.652895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5604.657512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5604.665167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5608.327869] Lustre: server umount lustre-OST0000 complete [ 5609.708447] LustreError: 30273:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5609.708597] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5609.709015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5609.709016] Lustre: Skipped 1 previous similar message [ 5609.720676] LustreError: 30273:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 5614.716745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5614.717031] LustreError: 29387:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5614.728535] Lustre: Skipped 4 previous similar messages [ 5615.747712] Lustre: server umount lustre-MDT0000 complete [ 5616.824475] LustreError: 27586:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734260563 with bad export cookie 13063265361671587060 [ 5616.825473] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5616.832808] LustreError: 27586:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5616.930173] Lustre: server umount lustre-MDT0001 complete [ 5619.943498] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 5632.938911] Lustre: server umount lustre-OST0001 complete [ 5634.398735] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 5639.047678] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5641.266268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5643.293169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5644.976100] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5644.982404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5646.610056] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5646.613035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5648.288722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5648.310825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5648.420336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5648.431061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5648.473652] Lustre: lustre-MDT0000: new disk, initializing [ 5648.494118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5648.499590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5649.529046] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5653.108952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5653.125998] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5653.150787] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5653.158558] Lustre: 5000:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5653.170118] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5653.172205] Lustre: Skipped 1 previous similar message [ 5653.204860] Lustre: lustre-MDT0001: new disk, initializing [ 5653.240892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5653.245224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5654.059283] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5656.608935] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5657.413550] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5659.123298] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5659.126297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5659.142677] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5659.145806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5659.209345] Lustre: 6195:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5659.219697] Lustre: lustre-OST0000: new disk, initializing [ 5659.221267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5659.237086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5659.240376] Lustre: Skipped 1 previous similar message [ 5660.489427] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5660.614215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5660.616694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5660.634149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5663.008779] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5663.795151] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5663.842113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5664.634871] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5664.695850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5665.629168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5665.636027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5665.645445] Lustre: Skipped 2 previous similar messages [ 5665.650064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5665.653646] Lustre: Skipped 1 previous similar message [ 5671.396059] Lustre: server umount lustre-OST0000 complete [ 5673.244722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5673.248846] LustreError: Skipped 1 previous similar message [ 5673.251473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.258911] Lustre: Skipped 1 previous similar message [ 5675.676319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5675.678045] Lustre: Skipped 4 previous similar messages [ 5678.523900] Lustre: server umount lustre-MDT0000 complete [ 5679.512881] LustreError: 4044:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734260625 with bad export cookie 13063265361671589069 [ 5679.515018] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5679.521924] LustreError: 4044:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5679.631062] Lustre: server umount lustre-MDT0001 complete [ 5681.955698] Lustre: DEBUG MARKER: == conf-sanity-1 test 132: hsm_actions processed after failover ========================================================== 06:03:47 (1734260627) [ 5685.673392] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 5687.632718] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5690.365084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5692.595185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5694.635700] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5694.642168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5696.508585] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5696.511677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5698.344266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5698.364083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.446769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5698.456206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5698.498948] Lustre: lustre-MDT0000: new disk, initializing [ 5698.517284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5698.523289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5699.509114] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5702.572370] Lustre: server umount lustre-MDT0000 complete [ 5703.538294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5705.363384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5705.379756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5705.453079] Lustre: Found index 0 for lustre-MDT0000, updating log [ 5706.245037] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5707.564884] Lustre: server umount lustre-MDT0000 complete [ 5709.536004] Lustre: DEBUG MARKER: == conf-sanity-1 test 133: stripe QOS: free space balance in a pool ========================================================== 06:04:15 (1734260655) [ 5709.913131] Lustre: DEBUG MARKER: SKIP: conf-sanity-1 test_133 needs >= 4 OSTs [ 5710.295056] Lustre: DEBUG MARKER: == conf-sanity-1 test 135: check the behavior when changelog is wrapped around ========================================================== 06:04:15 (1734260655) [ 5714.354728] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 5715.930181] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5718.664949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5720.658178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5722.586533] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5722.589612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5724.527124] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5724.533258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5726.445721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5726.464593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5726.537675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5726.539581] Lustre: Skipped 2 previous similar messages [ 5726.547454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5726.549417] Lustre: Skipped 1 previous similar message [ 5726.590604] Lustre: lustre-MDT0000: new disk, initializing [ 5726.616701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5727.609273] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5731.124541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5731.142008] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5731.175754] Lustre: 18680:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5731.267934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5731.271689] Lustre: Skipped 2 previous similar messages [ 5731.286048] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5732.259345] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5735.040360] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5736.057318] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5738.023050] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5738.027435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5738.047675] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5738.051338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5739.486691] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5739.768251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5739.792398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5742.294375] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5745.833714] Lustre: lustre-MDD0000: changelog on [ 5746.761894] Lustre: lustre-MDD0001: changelog on [ 5799.330228] Lustre: 17744:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5813.518064] Lustre: lustre-MDD0001: changelog off [ 5814.508269] Lustre: lustre-MDD0000: changelog off [ 5819.901130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.901734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5819.902004] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5819.909214] Lustre: Skipped 2 previous similar messages [ 5821.941294] Lustre: server umount lustre-OST0000 complete [ 5829.274102] Lustre: server umount lustre-MDT0000 complete [ 5829.916450] LustreError: 18710:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5829.921352] LustreError: 18710:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 5830.340712] LustreError: 19903:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734260776 with bad export cookie 13063265361671590455 [ 5830.342793] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5830.353100] LustreError: 19903:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5830.466858] Lustre: server umount lustre-MDT0001 complete [ 5833.876249] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 5834.448143] Key type lgssc unregistered [ 5834.536487] LNet: 24528:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5835.546274] LNet: Removed LNI 192.168.204.130@tcp [ 5840.804545] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 5845.568108] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5845.680744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5845.716305] alg: No test for adler32 (adler32-zlib) [ 5846.540209] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 5846.588753] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 5846.590454] LNet: Accept secure, port 988 [ 5848.118964] Key type lgssc registered [ 5848.426590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5850.955372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5852.707551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5854.345228] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5854.349079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5856.076904] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5856.080629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5858.242209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5858.263784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5858.271754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5859.358161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5859.367955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5859.419225] Lustre: lustre-MDT0000: new disk, initializing [ 5859.446533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5859.458706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5860.368649] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5864.207483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5864.229546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5864.254539] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5864.261024] Lustre: 29261:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5864.271580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5864.273117] Lustre: Skipped 1 previous similar message [ 5864.303242] Lustre: lustre-MDT0001: new disk, initializing [ 5864.323402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5864.332183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5864.336437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5865.296515] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5867.985123] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5868.730088] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5871.080474] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5871.087035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5871.108938] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5871.116365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5871.200717] Lustre: 30458:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5871.213296] Lustre: lustre-OST0000: new disk, initializing [ 5871.216932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5871.241182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5872.809672] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5873.352117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5873.354340] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5873.368781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5875.264483] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5877.088610] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_module obdecho/obdecho [ 5877.703679] LustreError: 31516:0:(echo_client.c:688:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5877.708669] LustreError: 31516:0:(obd_config.c:797:class_setup()) setup ec failed (-22) [ 5878.313148] LustreError: 31644:0:(echo_client.c:1894:echo_md_handler()) Can not resolve the path : rc = -2 [ 5883.388737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.389261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5883.401969] Lustre: Skipped 2 previous similar messages [ 5887.089076] Lustre: server umount lustre-MDT0000 complete [ 5888.141735] LustreError: 29275:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734260834 with bad export cookie 12526446281538197344 [ 5888.142881] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5888.152105] LustreError: 29275:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5888.243171] Lustre: server umount lustre-MDT0001 complete [ 5899.431033] Lustre: server umount lustre-OST0000 complete [ 5904.861835] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 5906.239392] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5908.246694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5910.244994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5911.690515] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5911.693705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5913.216103] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5913.219898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5915.397563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5915.414250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5915.502373] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5915.515097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5915.570026] Lustre: lustre-MDT0000: new disk, initializing [ 5915.601274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5915.607985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5916.528087] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5920.232120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5920.249557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5920.269323] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5920.275374] Lustre: 4827:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5920.284139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5920.285623] Lustre: Skipped 1 previous similar message [ 5920.316738] Lustre: lustre-MDT0001: new disk, initializing [ 5920.336668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5920.345297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5920.350213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5921.211995] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5923.845008] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5924.964598] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5926.866055] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5926.869097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5926.885491] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5926.888059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5926.949474] Lustre: 6023:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5928.529836] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5931.110180] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5931.943773] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5933.215280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5933.225733] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5933.252612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5934.055411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5935.026565] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5935.069486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5938.253577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5938.253906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5938.254290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5938.254293] Lustre: Skipped 2 previous similar messages [ 5938.278827] Lustre: Skipped 1 previous similar message [ 5941.623855] Lustre: server umount lustre-OST0000 complete [ 5943.260674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5943.261256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5943.261258] Lustre: Skipped 1 previous similar message [ 5943.270775] Lustre: Skipped 1 previous similar message [ 5948.268357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5948.271708] Lustre: Skipped 2 previous similar messages [ 5948.829507] Lustre: server umount lustre-MDT0000 complete [ 5949.988815] LustreError: 3870:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734260896 with bad export cookie 12526446281538198289 [ 5949.991297] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5949.999811] LustreError: 3870:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5950.114340] Lustre: server umount lustre-MDT0001 complete [ 5951.215177] Lustre: DEBUG MARKER: == conf-sanity-1 test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:08:16 (1734260896) [ 5951.713829] Lustre: DEBUG MARKER: SKIP: conf-sanity-1 test_137 needs >= 3 MDTs [ 5952.191977] Lustre: DEBUG MARKER: == conf-sanity-1 test 140: remove_updatelog script actions ========================================================== 06:08:17 (1734260897) [ 5954.579746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5954.762124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5954.764739] Lustre: Skipped 1 previous similar message [ 5955.665615] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5958.284884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5958.337979] Lustre: 9728:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5959.405351] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5961.014402] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5961.979979] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5964.280116] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5964.286562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5966.306259] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5968.279777] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5978.455272] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (stopping) [ 5978.460542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5978.461491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5978.461492] LustreError: Skipped 1 previous similar message [ 5978.469059] Lustre: Skipped 2 previous similar messages [ 5978.666962] Lustre: server umount lustre-MDT0000 complete [ 5979.751479] LustreError: 8847:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734260926 with bad export cookie 12526446281538199157 [ 5979.752625] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5979.757210] LustreError: 8847:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5983.455165] LustreError: 11538:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5983.461967] LustreError: 11538:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 5983.468604] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5984.470387] LustreError: 11538:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5984.475487] LustreError: 11538:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 5985.891202] Lustre: server umount lustre-MDT0001 complete [ 5987.714907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5991.484403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5991.591386] LustreError: 13300:0:(llog.c:1491:llog_backup()) MGC192.168.204.130@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5991.596480] Lustre: 13300:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.204.130@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6003.503124] LustreError: 13307:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6008.504327] LustreError: 13307:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6008.514293] LustreError: 13307:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 6014.487557] LustreError: 13336:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6014.494962] LustreError: 13336:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 5 previous similar messages [ 6024.508632] LustreError: 13336:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6024.517476] LustreError: 13336:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 7 previous similar messages [ 6041.692259] LustreError: 13306:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6041.696711] LustreError: 13306:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 13 previous similar messages [ 6051.667253] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6051.672607] Lustre: 13300:0:(mgc_request.c:1858:mgc_process_log()) MGC192.168.204.130@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6074.607251] LustreError: 13336:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6074.612489] LustreError: 13336:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 18 previous similar messages [ 6078.484217] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6078.488505] Lustre: 8857:0:(mgc_request.c:1858:mgc_process_log()) MGC192.168.204.130@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6091.692299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6091.697694] Lustre: Skipped 2 previous similar messages [ 6092.606820] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6094.948687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6096.703287] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 6098.494172] LustreError: 8857:0:(mgc_request.c:616:do_requeue()) failed processing log: -5 [ 6099.707279] LustreError: 26121:0:(client.c:1310:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800a4dd6d80 x1818504244328320/t0(0) o250->MGC192.168.204.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 6100.730188] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6102.353746] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6104.782303] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 6105.975628] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6106.988994] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6110.481749] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6119.805172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6119.805198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6119.808624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6119.808626] Lustre: Skipped 6 previous similar messages [ 6119.829740] Lustre: Skipped 2 previous similar messages [ 6122.300180] Lustre: server umount lustre-MDT0000 complete [ 6123.501532] LustreError: 8886:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734261069 with bad export cookie 12526446281538205618 [ 6123.502536] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6123.506716] LustreError: 8886:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6123.616420] Lustre: server umount lustre-MDT0001 complete [ 6134.999578] Lustre: server umount lustre-OST0000 complete [ 6141.305569] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 6143.192643] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6145.999091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6147.999005] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6150.193516] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6150.200855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6152.447530] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6152.456071] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6154.688831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6154.709966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.777337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6154.785211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6154.787531] Lustre: Skipped 1 previous similar message [ 6154.820947] Lustre: lustre-MDT0000: new disk, initializing [ 6154.822585] Lustre: Skipped 1 previous similar message [ 6154.839999] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6154.842715] Lustre: Skipped 1 previous similar message [ 6154.847209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6155.554832] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6159.539822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6159.565613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6159.613128] Lustre: 20338:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6159.619540] Lustre: 20338:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6159.730090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6160.622673] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6163.451377] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6164.573055] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6166.757003] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6166.762103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6166.788302] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6166.793940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6168.673109] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6169.074781] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6169.091674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6171.655716] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6172.746014] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6172.795588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6173.883556] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6173.958174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6179.100962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6179.101910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6179.102543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6179.102545] Lustre: Skipped 2 previous similar messages [ 6179.120353] LustreError: Skipped 1 previous similar message [ 6180.566739] Lustre: server umount lustre-OST0000 complete [ 6187.891690] Lustre: server umount lustre-MDT0000 complete [ 6189.083119] LustreError: 19393:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734261135 with bad export cookie 12526446281538222404 [ 6189.084946] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6189.097104] LustreError: 19393:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6189.240218] Lustre: server umount lustre-MDT0001 complete [ 6191.329493] Lustre: DEBUG MARKER: == conf-sanity-1 test 151a: damaged local config doesn't prevent mounting ========================================================== 06:12:16 (1734261136) [ 6196.287328] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6196.930633] Key type lgssc unregistered [ 6197.029404] LNet: 24879:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6198.037422] LNet: Removed LNI 192.168.204.130@tcp [ 6203.756297] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6203.954356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6203.961328] alg: No test for adler32 (adler32-zlib) [ 6204.772552] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 6204.822059] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 6204.823746] LNet: Accept secure, port 988 [ 6206.355985] Key type lgssc registered [ 6206.600297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6208.814358] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6208.819982] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6208.823054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6225.881033] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6231.883103] LustreError: 26175:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.130@tcp: failed processing log, type 4: rc = -110 [ 6237.889287] LustreError: 26175:0:(llog_osd.c:268:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x2:0x0] header magic: 0x0 (expected 0x10645539) [ 6237.896150] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6237.899553] LustreError: MGC192.168.204.130@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6237.906229] LustreError: 26175:0:(tgt_mount.c:1493:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6237.911491] LustreError: 26175:0:(tgt_mount.c:2222:server_fill_super()) Unable to start targets: -22 [ 6237.915244] LustreError: 26175:0:(tgt_mount.c:1721:server_put_super()) no obd lustre-OST0000 [ 6237.977649] Lustre: server umount lustre-OST0000 complete [ 6237.978906] LustreError: 26175:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6240.975206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6242.112318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6243.088784] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6245.848076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6245.892234] Lustre: 27699:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6245.978840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6247.051324] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6248.714549] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6249.909091] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6252.228602] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6252.234460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6252.324913] Lustre: 28812:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6252.336270] LustreError: 28812:0:(llog_osd.c:268:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x2:0x0] header magic: 0x0 (expected 0x10645539) [ 6252.343287] Lustre: 28812:0:(mgc_request_server.c:539:mgc_llog_local_copy()) MGC192.168.204.130@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6252.377231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6253.649763] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6255.488003] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6257.214600] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6257.215776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6257.229440] Lustre: Skipped 2 previous similar messages [ 6262.220494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6262.222044] Lustre: Skipped 4 previous similar messages [ 6262.290353] Lustre: server umount lustre-MDT0000 complete [ 6263.235114] LustreError: 26818:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734261209 with bad export cookie 1959919820864822083 [ 6263.237135] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6263.241989] LustreError: 27709:0:(client.c:1300:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88009284fb80 x1818504620689536/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 6263.258327] LustreError: 26818:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6263.383555] Lustre: server umount lustre-MDT0001 complete [ 6274.532082] Lustre: server umount lustre-OST0000 complete [ 6277.118094] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 6279.424319] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6282.519075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6284.770598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6286.729879] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6286.732393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6288.576600] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6288.579545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6291.050375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6291.078632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6291.178178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6291.186351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6291.234305] Lustre: lustre-MDT0000: new disk, initializing [ 6291.257769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6291.264553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6292.282908] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6295.904839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6295.920250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6295.939846] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6295.944779] Lustre: 1515:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6295.953180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6295.954558] Lustre: Skipped 1 previous similar message [ 6295.986103] Lustre: lustre-MDT0001: new disk, initializing [ 6296.004032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6296.013143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6296.015347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6296.857937] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6300.041735] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6301.023449] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6303.271466] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6303.277509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6303.304117] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6303.310882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6303.402331] Lustre: 2713:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6303.414323] Lustre: lustre-OST0000: new disk, initializing [ 6303.416536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6304.996556] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6307.057659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6307.063796] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6307.083853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6307.758365] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6308.829447] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6308.913935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6310.069588] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6310.156651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6312.076976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6312.077433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6312.078154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6313.005180] Lustre: server umount lustre-OST0000 complete [ 6316.028854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6316.031270] LustreError: Skipped 1 previous similar message [ 6316.032612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6316.037050] Lustre: Skipped 1 previous similar message [ 6316.039781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6316.042094] Lustre: Skipped 1 previous similar message [ 6320.421348] Lustre: server umount lustre-MDT0000 complete [ 6321.620074] LustreError: 2738:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734261267 with bad export cookie 1959919820864822769 [ 6321.622104] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6321.628124] LustreError: 2738:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6321.760502] Lustre: server umount lustre-MDT0001 complete [ 6324.434474] Lustre: DEBUG MARKER: == conf-sanity-1 test 151b: -ENOSPC doesn't affect mount ========================================================== 06:14:29 (1734261269) [ 6329.167349] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6329.946929] Key type lgssc unregistered [ 6330.055541] LNet: 6063:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6331.063361] LNet: Removed LNI 192.168.204.130@tcp [ 6336.635135] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6336.780605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6336.803552] alg: No test for adler32 (adler32-zlib) [ 6337.624664] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 6337.690641] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 6337.693704] LNet: Accept secure, port 988 [ 6339.236948] Key type lgssc registered [ 6339.458008] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6341.800628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6341.811818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6342.963866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6344.078653] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6347.083310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6347.130382] Lustre: 8182:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6347.249991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6348.468403] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6350.025872] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6351.079725] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6353.427818] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6353.430824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6353.496515] Lustre: 9361:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6353.503973] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6353.506221] LustreError: 9361:0:(llog.c:1531:llog_backup()) MGC192.168.204.130@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6353.510522] Lustre: 9361:0:(mgc_request_server.c:536:mgc_llog_local_copy()) MGC192.168.204.130@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6353.541489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6355.127127] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6356.828683] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6357.482408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6357.490484] Lustre: Skipped 1 previous similar message [ 6357.491066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6358.499448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6358.500103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6358.500104] Lustre: Skipped 1 previous similar message [ 6363.500860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6363.504661] Lustre: Skipped 4 previous similar messages [ 6363.575040] Lustre: server umount lustre-MDT0000 complete [ 6364.797283] LustreError: 7302:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734261311 with bad export cookie 6475685815137738360 [ 6364.799168] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6364.816488] LustreError: 7302:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6366.541086] LustreError: 8192:0:(client.c:1300:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880093330380 x1818504759101568/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 6366.641642] Lustre: server umount lustre-MDT0001 complete [ 6377.909079] Lustre: server umount lustre-OST0000 complete [ 6380.500132] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 6382.108723] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6384.569757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6386.423202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6388.431666] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6388.437689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6390.020258] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6390.023165] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6392.156402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6392.176986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6392.263197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6392.275245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6392.325005] Lustre: lustre-MDT0000: new disk, initializing [ 6392.346969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6392.354718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6393.233990] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6397.183182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6397.205788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6397.230376] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6397.236356] Lustre: 14502:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6397.243364] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6397.244357] Lustre: Skipped 1 previous similar message [ 6397.245455] LustreError: 14502:0:(llog.c:1531:llog_backup()) MGC192.168.204.130@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6397.248003] LustreError: 14502:0:(llog.c:1531:llog_backup()) Skipped 1 previous similar message [ 6397.249781] Lustre: 14502:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.204.130@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6397.255793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6397.257235] Lustre: Skipped 1 previous similar message [ 6397.289570] Lustre: lustre-MDT0001: new disk, initializing [ 6397.311285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6397.319822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6397.322681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6398.016365] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6400.685886] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6401.825899] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6404.290951] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6404.296918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6404.317078] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6404.322571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6404.434336] Lustre: 15701:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6404.447180] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6404.449881] Lustre: Skipped 1 previous similar message [ 6404.452143] LustreError: 15701:0:(llog.c:1531:llog_backup()) MGC192.168.204.130@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6404.457932] LustreError: 15701:0:(llog.c:1531:llog_backup()) Skipped 1 previous similar message [ 6404.462939] Lustre: 15701:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.204.130@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6404.470783] Lustre: 15701:0:(mgc_request_server.c:558:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6404.482491] Lustre: lustre-OST0000: new disk, initializing [ 6404.485283] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6406.146932] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6408.662091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6408.665989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6408.688612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6409.213480] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6410.320925] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6410.402182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6411.486912] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6411.530315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6413.676446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6413.678586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6413.683635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6418.073503] Lustre: server umount lustre-OST0000 complete [ 6422.348758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6422.355305] LustreError: Skipped 1 previous similar message [ 6422.358700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6422.365002] Lustre: Skipped 1 previous similar message [ 6422.367706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6422.370431] Lustre: Skipped 1 previous similar message [ 6425.494371] Lustre: server umount lustre-MDT0000 complete [ 6426.459889] LustreError: 14517:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734261372 with bad export cookie 6475685815137739060 [ 6426.461255] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6426.468350] LustreError: 14517:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6426.573663] Lustre: server umount lustre-MDT0001 complete [ 6429.253824] Lustre: DEBUG MARKER: == conf-sanity-1 test 153a: bypass invalid NIDs quickly == 06:16:14 (1734261374) [ 6433.280046] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 6435.068714] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6437.942176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6439.977130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6441.852594] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6441.856068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6443.660377] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6443.666090] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6445.941187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6445.967749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6446.097353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6446.111608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6446.171354] Lustre: lustre-MDT0000: new disk, initializing [ 6446.206020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6446.210488] Lustre: Skipped 1 previous similar message [ 6446.220576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6447.419664] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6451.894438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6451.923628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6451.965535] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6451.979618] Lustre: 22301:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6451.997132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6452.000442] Lustre: Skipped 1 previous similar message [ 6452.050781] Lustre: lustre-MDT0001: new disk, initializing [ 6452.098613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6452.104667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6453.274818] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6456.371994] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6457.310032] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6459.509374] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6459.516970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6459.538122] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6459.544199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6460.091080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6460.108692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6461.313137] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6464.371422] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6465.422640] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6465.511566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6466.580664] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6466.643820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6470.109174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6470.109287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6470.109563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6470.109565] Lustre: Skipped 2 previous similar messages [ 6470.126081] Lustre: Skipped 2 previous similar messages [ 6473.313220] Lustre: server umount lustre-OST0000 complete [ 6480.504046] Lustre: server umount lustre-MDT0000 complete [ 6481.906873] LustreError: 22315:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734261428 with bad export cookie 6475685815137739928 [ 6481.908296] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6481.914085] LustreError: 22315:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6482.032312] Lustre: server umount lustre-MDT0001 complete [ 6484.879736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.064568] LustreError: 26061:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6485.073031] LustreError: 26061:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 6485.085733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6485.088532] Lustre: Skipped 2 previous similar messages [ 6486.162712] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6489.281818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6489.329430] Lustre: 26922:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6489.336180] Lustre: 26922:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6490.512062] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6492.226233] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6493.355781] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6495.559977] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6495.565837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6497.314294] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6498.758867] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6524.606244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6524.611821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6524.620566] Lustre: Skipped 2 previous similar messages [ 6524.623953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6524.628276] Lustre: Skipped 5 previous similar messages [ 6526.236789] Lustre: server umount lustre-OST0000 complete [ 6529.468925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6533.751710] Lustre: server umount lustre-MDT0000 complete [ 6534.732607] LustreError: 26931:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6534.741421] LustreError: 26931:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 6535.021453] LustreError: 26043:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734261481 with bad export cookie 6475685815137740796 [ 6535.023989] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6535.034053] LustreError: 26043:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6535.175966] Lustre: server umount lustre-MDT0001 complete [ 6627.951430] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6628.846038] Key type lgssc unregistered [ 6628.956283] LNet: 30122:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6629.964268] LNet: Removed LNI 192.168.204.130@tcp [ 6638.072221] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 6643.256193] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6643.403913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6643.407468] alg: No test for adler32 (adler32-zlib) [ 6644.229087] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 6644.294698] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 6644.297748] LNet: Accept secure, port 988 [ 6645.842959] Key type lgssc registered [ 6646.171041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6649.226392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6651.590166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6653.596217] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6653.598900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6655.730976] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6655.734461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6658.059086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6658.083119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6658.090853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.173487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6659.182898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6659.225300] Lustre: lustre-MDT0000: new disk, initializing [ 6659.245921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6659.252029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6660.267616] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6664.492076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6664.513731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6664.557851] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6664.569468] Lustre: 2632:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6664.591852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6664.595962] Lustre: Skipped 1 previous similar message [ 6664.647395] Lustre: lustre-MDT0001: new disk, initializing [ 6664.682122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6664.698675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6664.703476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6665.859188] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6668.994044] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6670.209291] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6672.624506] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6672.629850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6672.653829] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6672.659364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6672.769966] Lustre: 3829:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6672.787584] Lustre: lustre-OST0000: new disk, initializing [ 6672.790927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6672.823476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6674.593401] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6677.640831] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6678.205574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6678.211142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6678.237492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6678.891241] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6678.972251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6679.904711] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6679.985835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6683.228686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6683.229537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6683.230151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6686.752797] Lustre: server umount lustre-OST0000 complete [ 6688.236724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6688.237437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6688.237439] Lustre: Skipped 1 previous similar message [ 6688.249031] Lustre: Skipped 2 previous similar messages [ 6693.244742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6693.249120] Lustre: Skipped 2 previous similar messages [ 6694.124139] Lustre: server umount lustre-MDT0000 complete [ 6695.550167] LustreError: 1683:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734261641 with bad export cookie 18001746209278522272 [ 6695.553151] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6695.561911] LustreError: 1683:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6695.708706] Lustre: server umount lustre-MDT0001 complete [ 6700.171437] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6702.899238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6703.098000] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6703.923302] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6706.708026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6706.749927] Lustre: 8036:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6707.827400] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6708.630286] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 6710.671042] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6710.677558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6710.777658] Lustre: 8958:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6710.821163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6710.824234] Lustre: Skipped 1 previous similar message [ 6712.616136] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6715.501099] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6715.504046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6715.522114] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6715.525343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6715.551571] Lustre: 9953:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6715.560515] Lustre: lustre-OST0001: new disk, initializing [ 6715.562178] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6716.926129] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6719.445335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6719.448694] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6719.461659] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6722.157104] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6725.438182] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 6734.211342] Lustre: DEBUG MARKER: == conf-sanity-1 test 153c: don't stuck on unreached NID ========================================================== 06:21:19 (1734261679) [ 6736.876779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6736.881061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6736.888969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6736.989977] Lustre: server umount lustre-MDT0000 complete [ 6738.317155] LustreError: 7156:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734261684 with bad export cookie 18001746209278523140 [ 6738.320291] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6738.331136] LustreError: 7156:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6738.459181] Lustre: server umount lustre-MDT0001 complete [ 6754.811954] Lustre: 31915:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734261685/real 1734261685] req@ffff88006efac000 x1818505081078528/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1734261701 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 6754.812035] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6754.832432] Lustre: 31915:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6758.803965] Lustre: 31918:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734261689/real 1734261689] req@ffff88006efaea00 x1818505081079040/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1734261705 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 6759.862448] Lustre: server umount lustre-OST0000 complete [ 6761.280948] Lustre: server umount lustre-OST0001 complete [ 6762.992197] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 6765.148907] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6768.029212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6770.058961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6771.881217] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6771.884855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6773.874731] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6773.879120] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6775.735769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6775.750631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6775.818958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6775.828718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6775.861698] Lustre: lustre-MDT0000: new disk, initializing [ 6775.878540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6775.881432] Lustre: Skipped 1 previous similar message [ 6775.886172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6776.603319] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6780.667503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6780.688701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6780.738122] Lustre: 15784:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6780.853621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6781.921011] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6785.049682] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6786.263950] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6788.751719] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6788.755639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6788.777992] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6788.784472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6788.899434] Lustre: lustre-OST0000: new disk, initializing [ 6788.901986] Lustre: Skipped 1 previous similar message [ 6788.905318] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6788.908397] Lustre: Skipped 2 previous similar messages [ 6790.427969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6790.430605] Lustre: Skipped 1 previous similar message [ 6790.433185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6790.453418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6790.716939] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6793.835008] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6794.928630] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6795.003241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6796.122530] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6796.178949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6800.461065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6800.461635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6800.462080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6800.473729] Lustre: Skipped 2 previous similar messages [ 6802.791619] Lustre: server umount lustre-OST0000 complete [ 6810.009395] Lustre: server umount lustre-MDT0000 complete [ 6810.476778] LustreError: 14856:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.486132] LustreError: 14856:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 6811.417213] LustreError: 17009:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734261757 with bad export cookie 18001746209278525240 [ 6811.419980] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6811.430075] LustreError: 17009:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6811.567633] Lustre: server umount lustre-MDT0001 complete [ 6814.631397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6814.760810] LustreError: 19546:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6814.780549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6814.782665] Lustre: Skipped 2 previous similar messages [ 6815.857017] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6818.810833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6818.859495] Lustre: 20407:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6818.864879] Lustre: 20407:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6819.920018] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6821.185274] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6822.259037] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6824.369173] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6824.375636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6826.008839] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6827.495904] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6970.748867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6970.753467] LustreError: Skipped 1 previous similar message [ 6970.756178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.763682] Lustre: Skipped 2 previous similar messages [ 6970.766982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6970.770657] Lustre: Skipped 3 previous similar messages [ 6974.749028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6976.392636] Lustre: server umount lustre-OST0000 complete [ 6983.859570] Lustre: server umount lustre-MDT0000 complete [ 6984.764688] LustreError: 19545:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6984.773540] LustreError: 19545:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 6985.261067] LustreError: 19569:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734261931 with bad export cookie 18001746209278526122 [ 6985.263536] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6985.272282] LustreError: 19569:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6985.431213] Lustre: server umount lustre-MDT0001 complete [ 7086.470420] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 7087.413652] Key type lgssc unregistered [ 7087.520370] LNet: 23587:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7088.528520] LNet: Removed LNI 192.168.204.130@tcp [ 7097.079439] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 7102.085687] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 7102.232752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7102.236293] alg: No test for adler32 (adler32-zlib) [ 7103.059250] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 7103.127511] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 7103.130680] LNet: Accept secure, port 988 [ 7104.676966] Key type lgssc registered [ 7105.035997] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7107.965273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7110.229944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7112.287923] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7112.291398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7114.258929] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 7114.266236] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7118.495563] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 7121.525374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7121.547059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7121.556813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7122.659733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7122.673556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7122.729612] Lustre: lustre-MDT0000: new disk, initializing [ 7122.760508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7122.771937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7123.775682] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 7128.144410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7128.170482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7128.212372] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7128.224229] Lustre: 29293:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7128.243596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7128.247381] Lustre: Skipped 1 previous similar message [ 7128.305189] Lustre: lustre-MDT0001: new disk, initializing [ 7128.339123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7128.357235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7128.363958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7129.509590] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 7131.850589] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 7134.215705] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7134.221447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7134.243347] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7134.248590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7134.343876] Lustre: 30299:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7134.358082] Lustre: lustre-OST0000: new disk, initializing [ 7134.361334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7134.397184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7136.133066] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 7136.789596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7136.795073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7136.829440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7140.469290] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 7140.475270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7140.498022] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 7140.503355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7140.556279] Lustre: 31378:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7140.575110] Lustre: lustre-OST0001: new disk, initializing [ 7140.578426] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7140.616275] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7142.389247] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 7145.662728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7145.667119] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7145.685048] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7147.633851] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7152.240167] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 7160.700539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7160.701165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7160.704881] Lustre: Skipped 3 previous similar messages [ 7165.708905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7165.713490] Lustre: Skipped 6 previous similar messages [ 7166.331426] Lustre: server umount lustre-MDT0000 complete [ 7167.604126] LustreError: 28343:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734262113 with bad export cookie 1871877530703155033 [ 7167.606682] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7167.617134] LustreError: 28343:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7167.726888] Lustre: server umount lustre-MDT0001 complete [ 7178.739710] Lustre: server umount lustre-OST0000 complete [ 7190.216344] Lustre: server umount lustre-OST0001 complete [ 7195.501288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7198.606690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7198.873846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7201.919080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7204.417218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7204.670195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7210.026874] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 7213.123776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7213.135488] Lustre: lustre-MDT0000: reset Object Index mappings [ 7213.377076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.393953] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 7217.485984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7217.493504] Lustre: lustre-MDT0001: reset Object Index mappings [ 7217.523381] Lustre: 6227:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7218.500222] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 7219.575304] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 7221.856163] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7221.860329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7221.929539] Lustre: 7161:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7221.962574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7221.965508] Lustre: Skipped 1 previous similar message [ 7222.962574] LustreError: 7168:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7222.964538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7222.980833] LustreError: 7168:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 7223.670456] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 7226.474553] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 7226.480327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7227.590083] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7228.311943] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 7231.520106] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7232.775294] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 7272.653575] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7272.654886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7272.666556] Lustre: Skipped 3 previous similar messages [ 7274.883156] Lustre: server umount lustre-MDT0000 complete [ 7276.062857] LustreError: 6696:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734262222 with bad export cookie 1871877530703161837 [ 7276.064124] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7276.069304] LustreError: 6696:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7276.174831] Lustre: server umount lustre-MDT0001 complete [ 7287.673653] Lustre: server umount lustre-OST0000 complete [ 7298.743385] Lustre: server umount lustre-OST0001 complete [ 7310.372483] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 7312.737968] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 7316.027217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7318.605104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7320.464888] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7320.469737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7322.658027] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 7322.661864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7323.038888] Lustre: DEBUG MARKER: == conf-sanity-1 test complete, duration 7139 sec ======== 06:31:08 (1734262268) [ 7323.605448] Lustre: DEBUG MARKER: === conf-sanity-1: start cleanup 06:31:08 (1734262268) === [ 7324.719196] Lustre: DEBUG MARKER: === conf-sanity-1: finish cleanup 06:31:10 (1734262270) === [ 7335.443445] device-mapper: core: cleaned up [ 7339.101984] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 7339.996030] Key type lgssc unregistered [ 7340.103533] LNet: 16729:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7341.111410] LNet: Removed LNI 192.168.204.130@tcp