Linux commands everyone should know

sysadmin commands for Linux development environments, containers, virtual machines (VMs), and bare metal.

In a world bursting with new tools and diverse development environments, it’s practically a necessity for any developer or engineer to learn some basic sysadmin commands. Specific commands and packages can help developers organize, troubleshoot, and optimize their applications and—when things go wrong—provide valuable triage information to operators and sysadmins.

Whether you are a new developer or want to manage your own application, the following basic sysadmin commands can help you better understand your applications. They can also help you describe problems to sysadmins troubleshooting why an application might work locally but not on a remote host. These commands apply to Linux development environments, containers, virtual machines (VMs), and bare metal.

1. curl

curl transfers a URL. Use this command to test an application’s endpoint or connectivity to an upstream service endpoint. curl can be useful for determining if your application can reach another service, such as a database, or checking if your service is healthy.

As an example, imagine your application throws an HTTP 500 error indicating it can’t reach a MongoDB database:

$ curl -I -s myapplication:5000 HTTP/1.0 500 INTERNAL SERVER ERROR

The -I option shows the header information and the -s option silences the response body. Checking the endpoint of your database from your local desktop:

$ curl -I -s database:27017 HTTP/1.0 200 OK

So what could be the problem? Check if your application can get to other places besides the database from the application host:

$ curl -I -s https://opensource.com HTTP/1.1 200 OK

That seems to be okay. Now try to reach the database from the application host. Your application is using the database’s hostname, so try that first:

$ curl database:27017 curl: (6) Couldn‘t resolve host ‘database

This indicates that your application cannot resolve the database because the URL of the database is unavailable or the host (container or VM) does not have a nameserver it can use to resolve the hostname.

2. python -m json.tool / jq

After you issue curl, the output of the API call may be difficult to read. Sometimes, you want to pretty-print the JSON output to find a specific entry. Python has a built-in JSON library that can help with this. You use python -m json.tool to indent and organize the JSON. To use Python’s JSON module, pipe the output of a JSON file into the python -m json.tool command.

$ cat test.json {“title”:“Person”,“type”:“object”,“properties”:{“firstName”:{“type”:“string”},“lastName”:{“type”:“string”},“age”:{“description”:“Age in years”,“type”:“integer”,“minimum”:0}},“required”:[“firstName”,“lastName”]}

To use the Python library, pipe the output to Python with the -m (module) option.

$ cat test.json | python -m json.tool {     “properties”: {         “age”: {             “description”: “Age in years”,             “minimum”: 0,             “type”: “integer”         },         “firstName”: {             “type”: “string”         },         “lastName”: {             “type”: “string”         }     },     “required”: [         “firstName”,         “lastName”     ],     “title”: “Person”,     “type”: “object” }

For more advanced JSON parsing, you can install jq. jq provides some options that extract specific values from the JSON input. To pretty-print like the Python module above, simply apply jq to the output.

$ cat test.json | jq {   “title”: “Person”,   “type”: “object”,   “properties”: {     “firstName”: {       “type”: “string”     },     “lastName”: {       “type”: “string”     },     “age”: {       “description”: “Age in years”,       “type”: “integer”,       “minimum”: 0     }   },   “required”: [     “firstName”,     “lastName”   ] }

3. ls

ls lists files in a directory. Sysadmins and developers issue this command quite often. In the container space, this command can help determine your container image’s directory and files. Besides looking up your files, ls can help you examine your permissions. In the example below, you can’t run myapp because of a permissions issue. When you check the permissions using ls -l, you realize that the permissions do not have an “x” in -rw-r–r–, which are read and write only.

$ ./myapp bash: ./myapp: Permission denied $ ls -l myapp -rw-r–r–. 1 root root 33 Jul 21 18:36 myapp

4. tail

tail displays the last part of a file. You usually don’t need every log line to troubleshoot. Instead, you want to check what your logs say about the most recent request to your application. For example, you can use tail to check what happens in the logs when you make a request to your Apache HTTP server.

example_tail.png

Use tail -f to follow Apache HTTP server logs and see the requests as they happen.

Use tail -f to follow Apache HTTP logs and see the requests as they happen.

The -f option indicates the “follow” option, which outputs the log lines as they are written to the file. The example has a background script that accesses the endpoint every few seconds and the log records the request. Instead of following the log in real time, you can also use tail to see the last 100 lines of the file with the -n option.

$ tail -n 100 /var/log/httpd/access_log

5. cat

cat concatenates and prints files. You might issue cat to check the contents of your dependencies file or to confirm the version of the application that you have already built locally.

$ cat requirements.txt flask flask_pymongo

The example above checks whether your Python Flask application has Flask listed as a dependency.

6. grep

grep searches file patterns. If you are looking for a specific pattern in the output of another command, grep highlights the relevant lines. Use this command for searching log files, specific processes, and more. If you want to see if Apache Tomcat starts up, you might become overwhelmed by the number of lines. By piping that output to the grep command, you isolate the lines that indicate server startup.

$ cat tomcat.log | grep org.apache.catalina.startup.Catalina.start 01-Jul-2017 18:03:47.542 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 681 ms

7. ps

ps shows process status. Use this command to determine a running application or confirm an expected process. For example, if you want to check for a running Tomcat web server, you use ps with its options to obtain the process ID of Tomcat.

$ ps -ef UID        PID  PPID  C STIME TTY          TIME CMD root         1     0  2 18:55 ?        00:00:02 /docker-java-home/jre/bi root        59     0  0 18:55 pts/0    00:00:00 /bin/sh root        75    59  0 18:57 pts/0    00:00:00 ps -ef

For even more legibility, use ps and pipe it to grep.

$ ps -ef | grep tomcat root         1     0  1 18:55 ?        00:00:02 /docker-java-home/jre/bi

8. env

env allows you to set or print the environment variables. During troubleshooting, you may find it useful for checking if the wrong environment variable prevents your application from starting. In the example below, this command is used to check the environment variables set on your application’s host.

$ env PYTHON_PIP_VERSION=9.0.1 HOME=/root DB_NAME=test PATH=/usr/local/bin:/usr/local/sbin LANG=C.UTF-8 PYTHON_VERSION=3.4.6 PWD=/ DB_URI=mongodb://database:27017/test

Notice that the application is using Python3 and has environment variables to connect to a MongoDB database.

9. top

top displays and updates sorted process information. Use this tool to determine which processes are running and how much memory and CPU they consume. A common case occurs when you run an application and it dies a minute later. First, you check the application’s return error, which is a memory error.

$ tail myapp.log Traceback (most recent call last): MemoryError

Is your application really out of memory? To confirm, use top to determine how much CPU and memory your application consumes. When issuing top, you notice a Python application using most of the CPU, with its memory usage climbing, and suspect it is your application. While it runs, you hit the “C” key to see the full command and reverse-engineer if the process is your application. It turns out to be your memory-intensive application (memeater.py). When your application has run out of memory, the system kills it with an out-of-memory (OOM) error.

example_top.png

Issuing top against an application that consumes all of its memory.

The memory and CPU usage of the application increases, eventually being OOM-killed.

example_topwithc.png

Pressing C while running top shows the full command

By hitting the “C” key, you can see the full command that started the application.

In addition to checking your own application, you can use top to debug other processes that utilize CPU or memory.

10. netstat

netstat shows the network status. This command shows network ports in use and their incoming connections. However, netstat does not come out-of-the-box on Linux. If you need to install it, you can find it in the net-tools package. As a developer who experiments locally or pushes an application to a host, you may receive an error that a port is already allocated or an address is already in use. Using netstat with protocol, process and port options demonstrates that Apache HTTP server already uses port 80 on the below host.

example_netstat.png

netstat verifies that Apache is running on port 80

Using netstat -tulpn shows that Apache already uses port 80 on this machine.

 

By 

Full article:

Source: 20 Linux commands every sysadmin should know | Opensource.com

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s