Introspection Fails During Baremetal Provisioning Due to the FMS Error

Problem

• During baremetal provisioning, Introspection fails due to the FMS Error.

Environment

• Platform9 Managed OpenStack - v8.0 and Higher
• Ironic Regions

Cause

• This is a known bug and is reported internally with ID: IAAS-10796
• In ironic-inspector logs (on Management Plane), check for the below logs.

  • The dnsmasq filter sync fails while fetching ironic_macs in dnsmasq driver due to 504 error.

ironic-api.log

[Timestamp] 25367 ERROR ironic_inspector.pxe_filter.base [-] The PXE filter DnsmasqFilter, state=initialized encountered an exception: HttpException: 504: Server Error for url: https://[DU-FQDN]/ironic/v1/ports?fields=address, 504 Gateway Time-out: nginx/1.18.0; resetting the filter: openstack.exceptions.HttpException: HttpException: 504: Server Error for url: https://[DU-FQDN]/ironic/v1/ports?fields=address, 504 Gateway Time-out: nginx/1.18.0

[Timestamp] 25367 DEBUG ironic_inspector.pxe_filter.base [-] Resetting the PXE filter driver DnsmasqFilter, state=initialized reset /opt/pf9/ironic-inspector/lib/python3.9/site-packages/ironic_inspector/pxe_filter/base.py:121

[Timestamp] 772 25367 WARNING ironic_inspector.pxe_filter.base [-] Filter driver DnsmasqFilter, state=uninitialized disabling periodic sync task because of an invalid state.: ironic_inspector.pxe_filter.base.InvalidFilterDriverState: The PXE filter driver DnsmasqFilter, state=uninitialized: my fsm encountered an exception: Can not transition from state 'uninitialized' on event 'sync' (no defined transition)

• The following introspection attempt after this fails due to the FSM error.

ironic-inspector.log

[Timestamp] 25367 INFO ironic_inspector.node_cache [-] Node [Node UUID] was created successfully

[TImestamp] 417 25367 ERROR ironic_inspector.node_cache [-] [node: [Node UUID] state processing] Processing the error event because of an exception <class 'ironic_inspector.pxe_filter.base.InvalidFilterDriverState'>: The PXE filter driver DnsmasqFilter, state=uninitialized: my fsm encountered an exception: Can not transition from state 'uninitialized' on event 'sync' (no defined transition) raised by ironic_inspector.process._process_node: ironic_inspector.pxe_filter.base.InvalidFilterDriverState: The PXE filter driver DnsmasqFilter, state=uninitialized: my fsm encountered an exception: Can not transition from state 'uninitialized' on event 'sync' (no defined transition)

Workaround

• Workaround is to add exception handling for the ironic API call.
• Refer - Do not reset FSM when ironic ports cant be queried.
• Patch the controller host with below fix:

On Controller Host

$ sudo vim /opt/pf9/ironic-inspector/lib/python3.9/site-packages/ironic_inspector/pxe_filter/dnsmasq.py +88

---
        try:
            # active_macs are the MACs for which introspection is active
            active_macs = node_cache.active_macs()
            # ironic_macs are all the MACs know to ironic (all ironic ports)
            ironic_macs = set(port.address for port in
                              ir_utils.call_with_retries(ironic.ports, limit=None,
                                                         fields=['address']))
        except Exception:
            LOG.exception(
                "Could not list ironic ports, can not sync dnsmasq PXE filter")
            return

• Restart Ironic services

Command

$ sudo systemctl restart pf9-ironic-*

Additional Information

• The issue is fixed in the Platform9 Managed Openstack v5.10.1 Release.