Author Archives: Kaven Gagnon

Getting the following error on your Juniper EX/MX/QFX virtual chassis?

fpc1 CMLC: Going disconnected; Routing engine chassis socket closed abruptly

This message is informational and does not necessary indicate any serious issue, for example in a graceful Routing Engine switchover (GRES) context / if you flipped the routing-engine (RE) on purpose. However if this is message is printed repeateadly, without any virtual chassis (VC) topology change or manual intervention, this may indicate a more serious issue that worth investigating.

The most common issue is cabling between your VC members, and most likely to happen if the members are distant from each others connected with SFP+/QSFP+ and fiber rather than the short length DAC cables (it does not mean that no issue can happen with DAC cables, it is just that the elements and distance in the chain of even is reduced).

Here are the common symptom/possibilities on the vc-ports :

• Defective SFP+/QSFP+ optic (dying optic, losing power transmission capability)
• Fiber length too tight for the optic capability (check laser transmit/receive power)
• Damaged fiber/connector, bad fusion point, dirty connector/optic (investigate with light testers, loopback, OTDR, clean the tips…)
• Port flapping (can be caused by all the above)
• If you are using DAC cables and observe CRCs, just swap it out with another one and monitor if there is any change in the situation.

OBSERVATION : When such issue occur and there is a lot of flaps/errors between VC ports members, you may observe a higher load on the CPU/RE than usual and some functionalities such as ICMP drop, SNMP polling issues. If your device is usually very busy and near capacity, more serious service impacting issues may occur on layer 3 services such as BGP and OSPF  as well (especially on EX series devices).

In most cases, you will observe Cyclic Redundancy Check (CRC) errors on the vc-port(s). You should check it out using the following command :

And investigate accordingly based on the tips provided above.

Here is an example output below of a VC with this behavior showing CRC errors (yeah… I knew the fiber that was delivered to me by the cabling technician was bad just by looking at it, I was told it has been tested – trust no one!) :

I ran into a weird issue trying to adopt several Ubiquiti UniFi AP Mesh Pro access points on a newly created site within the UniFi controller lately. Here is the behavior encountered :

– Newly UniFi access point deployed showed up in the controller for adoption ;
– Clicking “Adopt” did started the provisioning process ;
– At the end of the provisioning (adoption), the access point show as “Disconnected”.

Additional information :

– APs were running firmware 3.7.58.6385 ;
– All three deployed APs ran into the same issue
– No filtering rules between APs and controller is blocking traffic.

Fix attempt :

1. Login to the AP using SSH and credentials (ubnt / ubnt) ;
2. Issue the command to inform the controller that a new access point is available :

This actually started the adoption and provisioning again, but still end up failing with “Disconnected” status.

So, what was the fix?

I did both steps mentioned above, but actually did the step two (2) repeatedly using the [up arrow] + [enter] key combination until I was kicked out from the SSH session (this meant that the provisioning process rebooted the AP to complete the adoption.

I believe this could have been caused by a bug in that particular firmware version that was on the APs initially.

If your Power over Ethernet (PoE) device(s) connected to an EX 4300-48p switch suddenly lose power unexpectedly after some time working just fine, you may be experiencing PoE controller software issue.

First of all, verify if you are not over your PoE power budget (this could be sometimes the case when you have a LOT of devices connected – this kind of situation occurs only if you have a lot of type 2 high powered PoE (IEEE 802.3at) devices connected to it) :

You should see a similar output like this :

You should look at the “Total PoE power consumed”. If you are equal or tight to the limit, it might be just your problem, you should disconnect a few devices/relocate them on another switch and see if there is any improvement.

In this particular situation, the output above was taken immediately after the “PoE outage” occurred. All devices were connected properly, but no power was injected by the switch and consumed by the devices. Down/Up the port(s) had no effect.

It then came to mind to have a look at the “PoE controller”

Here is an example of the output of this command :

As we can see, there is a notification that a new version of the PoE software is available. In my case, that was the issue, most likely a bug in the current version that I was running.

NOTE : It may be a good idea to validate first if you are running the recommended JunOS version by JTAC. If not, upgrading that first is recommended.

NOTICE : Before you continue, this upgrade need to be performed within a maintenance window, a minimum of 10 – 15 minutes will be required, as in addition of upgrading the PoE software, the switch will automatically reboot once completed. The switch may become unstable while performing the upgrade.

In order to perform the PoE software upgrade, do the following command :

NOTE : Please note that if you are running a virtual-chassis, the command will need to be performed for all unit and change the “fpc-slot” number matching the other units in the VC.

You will now notice the following output once you’ve issued the PoE firmware upgrade :

Once the switch(es) completed their upgrade and rebooted, the PoE injection should be stable and steady.

[This article may also apply to EX2200, EX3200, EX2300, EX3300, EX3400-p chassis as well]

If you want your SRX firewall to handle DNS requests on your network, you need to configure the forwarders to make this possible, in addition to a few other parameters.

First, make sure you have no local forwarders set on the device itself as it cannot be used along with the dns-proxy service – if you have any configured, they should be all removed :

Then, follow the step-by-step procedure below :

1. Configure the DNS proxy setting on the desired interface(s) where it should listen for DNS requests :

2. Configure the DNS resolver(s) where the requests will be resolved from (aka your ISPs or any public DNS service) :

3. Allow DNS traffic on the security zone :

4. Apply the configuration (use “commit synchronize” if you are running HA) :

Here is a sample of how it would look like :