Linux Device Troubleshooting Guide

The first thing you should always do when troubleshooting a device running a Linux OS is to try establishing a connection to it via Device Management.

What questions should I be asking?

How is it not working? What is the problem?

• Error message?
• See error message section below for details
• The system is running slowly/Poor application performance
• Can you establish a connection?
• No – investigate network connectivity. FullCount Support page (CTRL + ALT + F9) is helpful here.
• Yes – Run some basic networking tests via the command line
• ping
• nslookup
• portal.fullcount.net is the domain for central customers
• Check Tracker page for domain hosted on remote servers (Host Location URL)
• Create a Redmine task and assign it to Systems. Order numbers and/or specific dates/times will be mandatory when working on an issue of this nature. Typically the best thing that can be done with a slow/unresponsive application though is capture ping and nslookup info during the call.
• Device will not power on
• Try multiple sources/connections involving power
• Power outlets
• Power cords
• Power supplies
• This should only be troubleshot for out-of-warranty All in One terminals. Inquire with an experienced employee if you think you must take troubleshooting to this extent.
• External batteries (DT Research tablets, DLI tablets)
• Docking stations (tablets)
• Look for any sign of life out of the device
• The customer may report the device has a black screen – is it a lit up black screen? Does the screen not light up at all?
• For the All in One terminals, does the FullCount logo light up? Can the receipt printer power on/off?
• Boots to a white FullCount screen
• This is very commonly the same symptom as Server Not Found. Server Not Found/networking troubleshooting applies here.
• Something is physically broken on the device
• It’s extremely unlikely there’s anything we can do about it. Inquire with an experienced employee to be certain.

What is the error message?

• Server Not Found/InternetDisconnectedError
• Did your connection request in Device Management succeed? Can you ping this device from another device? Did pressing Try Again help? If not, you probably have an internet connectivity issue. FullCount Support page is helpful here.
• Are all devices getting this message? It’s likely a network-wide failure if so.
• Is a single terminal getting this message while others are not? It’s likely location-related, try swapping the location of the problem terminal with a working one to see where the problem goes.
• Name Not Resolved
• Even though it's not exactly the same thing, treat this the same way you would troubleshoot Server Not Found (information above).
• This error is specifically DNS-related, however, it could occur with a shotty network connection as well so don't forget to ping in conjunction with nslookup.
• System Encountered Unexpected Error…
• This is very likely a connectivity issue, perform basic network connectivity tests from the shell. See above for more details on what to do when the application is performing poorly.
• Unable to Connect
• Is this a remotely hosted customer?
• Are you able to establish a connection to the remote server?
• No – Investigate the remote server first (powered on, network connection, etc)
• Yes – Can you load the Back Office or Touchscreen application without issue?
• System/boot error – white text on a black screen?
• Instruct the customer to take a picture of the screen and send it to support@fullcount.net. Create a Redmine task and assign it to Systems or see if an experienced Support member can diagnose.

What available tools can be used to diagnose the issue?

CTRL + ALT + F9 (keyboard plugged into the device locally) – This is the most powerful tool you have at your disposal if you cannot establish a connection to the device. This keyboard shortcut displays all network information the device is currently configured with.

• Is the information on this screen blank? If so, the device has no network connectivity.
• If the device displays network information, notably an IP address, you can use that to try SSHing into the device. More on that below.

systemctl restart display-manager – restarts the application

It’s always useful to be able to restart the application without having to reboot the whole device. If a change has been made and you’d like to see if FullCount loads correctly give this a try.

reboot – reboots the device. This will kill your connection to the device.

ping command – sends packets of data to another host to elicit a response

This is a basic network communication test that FullCount Support should always exercise when troubleshooting an application problem – this is because pinging the host location URL gives us an idea of how well a device can communicate with our application or another host. Anything greater than 0% packet loss can be concerning as well as spikes in packet latency.

To execute this command, simply type ping followed by a destination host (a printer, a terminal, our domain, etc).

Examples:

ping 192.168.11.20

ping google.com

Here is an example of a ping test getting a response back from its destination:

If you are unable to communicate with the host you’re sending packets to, you might see the command “hang” or give this output:

Keep an eye on the last column of each received packet – this tells us how long the packet took to get to its destination and back. If you see sustained spikes of +200 ms then that could indicate sluggish/slow application.

nslookup command – queries internet domain name servers

All of our POS devices will require the use of a DNS server to run the FullCount Touchscreen because we are a web-based application. Without functioning DNS the application will not work because it will not be able to find our application on the web. Most equipment will be set up with redundant DNS configurations so when the primary DNS server fails the secondary DNS will kick in. This seems great, however, it is not uncommon to experience application issues when only the primary DNS server is down anyway. Furthermore, if the secondary DNS server is carrying all the weight then you can expect a slow/sluggish application because the primary DNS will continue to be used and will always fail before the secondary DNS steps in.

To execute this command, simply type nslookup followed by the domain you are trying to resolve. If you’d like to specify which DNS server you’re using for the query then add the IP address of that server after the domain.

Examples:

nslookup google.com

nslookup moen.com

nslookup portal.fullcount.net 8.8.8.8

Here is an example of positive standard output for nslookup:

The DNS server resolves the domain portal.fullcount.net to the address of 208.69.145.98, which is correct. This output also tells us that the server used for the query was 192.168.11.2 which is the primary DNS of the device used in the screenshot – everything here looks good.

Keep an eye out for standard output where the secondary DNS is the server that was used for the query – this would likely mean the primary DNS server is failing. In this scenario, you could try nslookup portal.fullcount.net [primary DNS] to see what results you get.

Here is an example of a standard error for nslookup:

This isn’t good. To verify we don’t have a hardware failure or local network issue you should try using Google’s public DNS, 8.8.8.8, to resolve portal.fullcount.net to ensure that the domain can be resolved from the device at all.

ssh – secure shell; establishes a secure tunnel between you and the host if you are attempting to connect to which allows for remote control of the device.

Device Management establishes SSH tunnels through a UI which makes it convenient to connect to a community’s devices. There are instances though where you might not be able to connect to a device via a requested connection but you are able to ping the device from a device that you did get a successful connection to. In these situations, try to SSH into the device using the ssh command. Simply enter the IP address of the host you want to connection and type yes followed by pressing Enter to get through.

arp -n - used to obtain MAC addresses

The arp command displays a table contains MAC addresses of recently pinged devices on the local area network. This command is helpful for checking if an IP conflict has occurred or simply verifying that the IP address you've pinged is the device you think it is.

arp -n