Q-SYS Networking and Topologies (Part 6)

Once all the other issues on the network are taken care of, we’re often left

with the third category of problem…network performance issues.

These are usually indicated by streaming faults and PTP clocking faults

in the Q-SYS eventlogs and device status blocks.

Remember the network requirements

for Q-LAN are very tight …280us end to end latency with 30us of allowable jitter.

We need audio packets to travel as quickly and consistently as possible.

When thinking about solving these problems, remember

that if the devices are not locked to the same PTP grandmaster,

you cannot hope to have consistent streaming.

Always concentrate on this problem first.

The important information for troubleshooting PTP problems is in the Core and peripheral status blocks.

The first thing of note is exactly what device is the PTP grandmaster.

If a Q-SYS device is named in the grandmaster field, then it’s easy to know which device that is.

If you see a MAC address instead, that would indicate that a 3rd party device is the grandmaster.

You could use the OUI of that device to find out exactly which device that is.

A good example would be the use of a 110f in conjunction with a SHURE mxa 910.

The Shure device uses a lower number for PTP priority 1, so it is elected the grandmaster by default.

You would see that device’s MAC in the grandmaster field for both the Core and peripheral in this case.

In general you want to first observe that the device serving as grandmaster makes sense.

Red PTP clocking errors will show up in the status bar at the top of the blocks.

Observe the Clock offset around the time these errors occur.

If you see sudden changes in this offset, especially if it’s going from a positive to a negative number abruptly,

this indicates jitter on the network.

Jitter is the enemy of PTP, so you’ll want to concentrate

on getting clock packets delivered consistently on the network.

This often means the proper QoS policies haven’t been put in place from end to end on the network.

When troubleshooting Q-LAN packet delivery problems,

it’s very useful to enable the verbose statistics in the device status blocks.

We’d often recommend this be done first thing, as it requires redeployment of the design to enable.

If you’re having problems that crop up only after some time,

you may have to wait a long time for the problem to resurface.

A Q-LAN stream consists of 3000 packets per second per stream, so it’s a lot to keep track of.

The verbose statistics can tell you exactly how many did or did not arrive,

whether they were completely missing or late.

Of course late packets would indicate excess latency on the network.

Missing packets indicate that packets are being dropped by the network.

You can also get a good idea here of how bad the problem is:

If you have one late packet per million, that’s something that users aren’t even likely to hear.

I you have high numbers of late packets that are rapidly counting up, it’s an issue you want to deal with quickly.

You might recall some of the scenarios that can get us into trouble from the previous trainings.

We often have isolated room systems connected to the corporate network for Q-SYS softphone registration,

which can cause all the individual room systems to try and sync to each other over a non-compliant network.

As long as these systems don’t need to share Q-LAN audio,

the solution here is to simply disable PTP on the LANB network ports.

This is done in the network services tab of Q-SYS Core Manager.

If the systems in question must share Q-LAN or AES67 audio,

they must be synchronized to a single PTP grandmaster, meaning they must also be on the same PTP domain.

It’s still valid to manipulate the PTP priority of one system over the other to get the right results.

For example, if one system is mission critical such as the bowl of a stadium seating area,

it makes sense to prioritize that Core as the PTP grandmaster.

In other cases, the grandmaster might be on the very edge of the network making it hard to sync to all the other edges.

Prioritizing a more centralized device to be grandmaster minimizes how far clock packets must travel,

so that can often be a big help.

Once all the basics are covered and there are still PTP problems,

then the switch configuration will need to be examined.

You’ll want to make sure the recommendations are being followed:

Make sure a strict priority queueing model is being used.

Make sure you know which DSCP value clock packets are using and that value is assigned to the

highest-priority queue. Remember sometimes that will be two values…56 for PTPv1 and 46 for PTPv2.

Make sure you know which DSCP value audio packets are using

and make sure that’s assigned the second highest priority queue.

Keep in mind these settings must be consistent on every switch in the network to know that traffic

will be prioritized properly.

This must be investigated for not just PTP clocking problems,

but for missing and late audio packet reports, too.

Here’s an illustration of these points in the setup for a commonly used switch: The DLink DGS1210-10P.

At the top left, we can see the switch is in DSCP QoS mode and using a strict priority queueing model.

This is exactly what we’d like. We have to look a little more closely to examine the queue assignments.

In most network switches the highest number queue represents the highest priority.

In this switch that’s queue 7. We see here that DSCP value 46 is assigned to that queue.

That would correspond to the PTPv2 DCSP marking in the Q-LAN QoS Preset.

We know the DSCP value associated with audio in that case is 34.

We see here that 34 is assigned to queue 6 as it should be. That's the second-highest queue.

Finally, DSCP value 26, which is Q-SYS video, is placed in queue 5.

Note that all other DSCP values are associated with priority zero, so they are not prioritized at all.

This is what we would expect in this situation

Remember these settings would be valid for the QLAN QoS preset in the Q-SYS design.

In the Audinate QoS Preset, we’d see 56 for queue 7, 46 for queue 6 and 26 for queue 5.

We also discussed the case with a mixture of new and old firmware Audinate devices where we have to account

for DSCP values 56, 46, 34 and 26 to make sure all traffic is properly prioritized.

This is what we’d expect in that case.

Going back to our troubleshooting mindset, let’s say you’ve looked at all your Q-SYS

and network configuration and it all looks ok.

You still have some late packets reported in your Q-SYS devices that you just can’t remedy.

There’s ONE last thing you can try.

Extending the Q-SYS network receive buffer in the Q-SYS Core properties.

This increases the overall audio latency by the amount your extend the buffer,

but it does make the Q-SYS network more tolerant to end to end latency.

This will not help, however, if your primary problem is network jitter.