This is a collection of some very useful command-line commands that eases the life of a DevOps Engineer.
This is a collection of some very useful command-line commands that eases the life of a DevOps Engineer.
Please Feel free to fork and/or PR if you have any additions.
Checking ports
netstat -tulpn
ss -ltp
ss -ltn
ss -stplu
lsof -i
lsof -ni | grep LISTEN
Linux Commands
cp -a /source/. /dest/
rm /path/to/dir/*
rm -r /path/to/dir/*
:%s/\<word\>\C/newword/g
Openssl
openssl s_client -connect google.com:443 -tls1_2
openssl req -out CSR.csr -new -newkey rsa:2048 -nodes -keyout privateKey.key
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout privateKey.key -out certificate.crt
openssl req -out CSR.csr -key privateKey.key -new
openssl x509 -x509toreq -in certificate.crt -out CSR.csr -signkey privateKey.key
openssl rsa -in privateKey.pem -out newPrivateKey.pem
openssl req -text -noout -verify -in CSR.csr
openssl rsa -in privateKey.key -check
openssl x509 -in certificate.crt -text -noout
openssl pkcs12 -info -in keyStore.p12
openssl x509 -inform der -in certificate.cer -out certificate.pem
openssl x509 -outform der -in certificate.pem -out certificate.der
openssl pkcs12 -in keyStore.pfx -out keyStore.pem -nodes
openssl pkcs12 -export -out certificate.pfx -inkey privateKey.key -in certificate.crt -certfile CACert.crt
awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' your_private_key.pem > output.txt
Listing Running Services Under SystemD in Linux
Check a public IP
curl http://whatismyip.org/
curl ifconfig.me
curl icanhazip.com
Return the IP of an interface
ifconfig en0 | grep --word-regexp inet | awk '{print $2}'
ip add show eth0 | awk '/inet/ {print $2}' | cut -d/ -f1 | head -1
ip -br a sh eth0 | awk '{ print $3 }'
(returns netmask)ip route show dev eth0 | awk '{print $7}'
hostname -I
(return ip only)Replace all occurrences of string in a directory
grep -rl "oldstring" ./ | xargs sed -i "" "s/oldstring/newstring/g"
Dig
dig <domain.com> @<ns-server>
dig <domain.com> ns
Disk checks
df -h | tail -n +2 | sort -rk5
du -h --max-depth=1 /tmp/
du -ah / | sort -n -r | head -n 50
du -sh *
du -h <dir> | grep '[0-9\.]\+G’
watch -n 10 df -ah
ncdu -q
du -x --max-depth=1|sort -rn|awk -F / -v c=$COLUMNS 'NR==1{t=$1} NR>1{r=int($1/t*c+.5); b="\033[1;31m"; for (i=0; i<r; i++) b=b"#"; printf " %5.2f%% %s\033[0m %s\n", $1/t*100, b, $2}'|tac
Docker
docker info
docker container ls -s
du -h --max-depth=1 /var/lib/docker
docker system df
docker volume ls
docker image ls
docker volume inspect VOLUME NAME
docker images | grep "<none>" | awk '{print $3}' | xargs docker rmi
docker rm $(docker ps -aq --filter status=exited)
docker rmi $(docker images -q --filter dangling=true)
docker volume rm $(docker volume ls -qf dangling=true)
docker system prune
curl -sSL https://get.docker.com/ubuntu/ | sudo sh
docker ps -q | xargs docker stats
docker logs --tail=300 -f <container_id>
docker build -t myimage:1.0 .
docker pull myimage:1.0
docker tag myimage:1.0 myrepo/myimage:2.0
docker push myrepo/myimage:2.0
docker container run --name web -p 5000:80 alpine:3.9
docker container stop web
docker container kill web
docker network ls
#Step1 - Save the Docker image as a tar file
docker save -o <path for generated tar file> <image name>
#Example
docker save -o c:/myfile.tar centos:16
#Step2 - copy your image to a new system with regular file transfer tools such as cp, scp or rsync(preferred for big files)
#Step3 - load the image into Docker
docker load -i <path to image tar file>
Shell Script to Install Docker on Ubuntu
#!/bin/bash
set -e
#Uninstall old versions
sudo apt-get remove docker docker-engine docker.io containerd runc
#Update the apt package index:
sudo apt-get update
#Install packages to allow apt to use a repository over HTTPS:
sudo apt-get install -y \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common
# Add docker's package signing key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
# Add repository
sudo add-apt-repository -y \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
# Install latest stable docker stable version
sudo apt-get update
sudo apt-get -y install docker-ce
# Enable & start docker
sudo systemctl enable docker
sudo systemctl start docker
# add current user to the docker group to avoid using sudo when running docker
sudo usermod -a -G docker $USER
# Output current version
docker -v
Shell Script to Install Docker on Centos
#!/bin/bash
#Get Docker Engine - Community for CentOS + docker compose
set -e
#Uninstall old versions
sudo yum remove docker docker-common docker-selinux docker-engine-selinux docker-engine docker-ce
#Update the packages:
sudo yum update -y
#Install needed packages
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
# Configure the docker-ce repo:
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# Install the latest docker-ce
sudo yum install docker-ce
# Enable & start docker
sudo systemctl enable docker.service
sudo systemctl start docker.service
# add current user to the docker group to avoid using sudo when running docker
sudo usermod -a -G docker $(whoami)
# Output current version
docker -v
Shell Script to Install Docker on AWS linux
#!/bin/bash
#Get Docker Engine - Community for CentOS + docker compose
set -e
#Uninstall old versions
sudo yum remove docker docker-common docker-selinux docker-engine-selinux docker-engine docker-ce
#Update the packages:
sudo yum update -y
#Install the most recent Docker Community Edition package.
sudo amazon-linux-extras install docker -y
# Enable & start docker
sudo service docker start
# add current user to the docker group to avoid using sudo when running docker
#sudo usermod -a -G docker ec2-user
sudo usermod -a -G docker $(whoami)
# Output current version
docker -v
Docker Compose
Shell Script to Install the latest version of docker-compose
#!/bin/bash
# get latest docker compose released tag
COMPOSE_VERSION=$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep 'tag_name' | cut -d\" -f4)
sudo curl -L "https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod a+x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
# Output the version
docker-compose -v
Dockerfile
FROM node:4.6
WORKDIR /app
ADD ./app
RUN npm install
EXPOSE 3000
CMD npm start
Find
find /tmp -not \( -path /tmp/dir -prune \) -type p -o -type b
Git
git rm $(git ls-files --deleted)
git reset --hard HEAD
git clean -xdf
git config --global http.sslVerify false
git pull --prune
git diff HEAD@{1} ../../production.hosts
git diff --cached <file>
git reset --soft HEAD~1
git log --stat
git pull --autostash
git commit --allow-empty -m "Trigger notification"
git remote set-url origin git://new.location"
Update .gitignore with the folder/file name you want to ignore. You can use anyone of the formats mentioned below (prefer format1)
### Format1 ###
node_modules/
node/
### Format2 ###
**/frontend/node_modules/**
**/frontend/node/**
Commit all the changes to git. Exclude the folder/files you dont want commit, in my case node_modules
Execute the following command to clear the cache
git rm -r --cached .
Execute git status command and it should output node_modules and sub directories marked for deletion
Now execute
git add .
git commit -m "fixed untracked files"
git push
Jenkins
#!/bin/bash
sudo yum update -y
sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat/jenkins.io.key
sudo yum install java-1.8.0 -y
sudo yum install jenkins -y
sudo service jenkins start
sudo cat
/var/lib/jenkins/secrets/initialAdminPassword
Grep
grep -R "foo” .
grep -i -C 10 "invalid view source” /var/log/info.log
grep -n “pattern” <file>
Iptables
iptables -nvL -t nat
Apache
Turn off “ServerSignature” and “ServerToken” on Apache
// Kali, Debian, Ubuntu Linux Mint
sudo vi /etc/apache2/apache2.conf
// CentOS, Fedora, RHEL , Arch Linux
sudo vi /etc/httpd/conf/httpd.conf
ServerSignature Off
ServerTokens Prod
// Kali, Debian, Ubuntu Linux Mint
sudo service apache2 restart
//Fedora, CentOS/RHEL 7,Arch Linux
systemctl restart httpd.service
Nginx
nginx -V
2>&1 nginx -V | xargs -n1
nginx -t
nginx -s stop
nginx -s start
nginx -s restart
nginx -s reload
Tomcat
Hide tomcat stack traces (showReport) as well as the Server info: Add below lines at the HOST section
<Valve className="org.apache.catalina.valves.ErrorReportValve"
showReport="false"
showServerInfo="false" />
Injecting HTTP Response with the secure header in Tomcat
<filter>
<filter-name>httpHeaderSecurity</filter-name>
<filter-class>org.apache.catalina.filters.HttpHeaderSecurityFilter</filter-class>
<async-supported>true</async-supported>
<init-param>
<param-name>antiClickJackingEnabled</param-name>
<param-value>true</param-value>
</init-param>
<init-param>
<param-name>antiClickJackingOption</param-name>
<param-value>DENY</param-value>
</init-param>
</filter>
<filter-mapping>
<filter-name>httpHeaderSecurity</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
<welcome-file-list>
<welcome-file>index.html</welcome-file>
</welcome-file-list>
Nmap
nmap -p <port> <host/IP>
nmap -sS <host/IP>
nmap --top-ports 10 <host/IP>
Password generation
openssl passwd -crypt <password>
makepasswd -count 1 -minchars 8
sudo htpasswd -c /etc/nginx/.htpasswd <user>
Removing files
find . -mtime +30 | xargs rm -rf
find . -type f -name "backup*" -mtime +7 -exec rm {} \;
SSH
ssh-keygen -q -t rsa -f ~/.ssh/<name> -N '' -C <name>
ssh-keygen -y -f eliarms.pem > eliarms.pub
Tail log with colored output
grc tail -f /var/log/filename
Tmux
tmux kill-window -t 0
tmux kill-window -t X
tmux new -s <name>
tmux ls
ps
ps auxwf
ps -efH
ps -elf | awk '{if ($5 == 1){print $4" "$5" "$15}}'
ps aux | grep Z
HSTS
Enable HTTP Strict Transport Security protocol in Tomcat To enable HSTS in Tomcat, follow these steps:
Open the <Tomcat>/conf/web.xml file in a text editor.
Uncomment the httpHeaderSecurity filter definition and the
<filter>
<filter-name>httpHeaderSecurity</filter-name>
<filter-class>org.apache.catalina.filters.HttpHeaderSecurityFilter</filter-class>
<init-param>
<param-name>hstsMaxAgeSeconds</param-name>
<param-value>31536000</param-value>
</init-param>
<async-supported>true</async-supported>
</filter>
<filter-mapping>
<filter-name>httpHeaderSecurity</filter-name>
<url-pattern>/*</url-pattern>
<dispatcher>REQUEST</dispatcher>
</filter-mapping>
Save the file
Restart Tomcat
Enable HTTP Strict Transport Security protocol in Apache To enable HSTS in Tomcat, follow these steps:
Open the <Apache>/conf/httpd.conf file in a text editor.
Uncomment the header module: LoadModule headers_module modules/mod_headers.so
Add a header setting in the VirtualHost section:
<VirtualHost www.example.com:80>
Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"
</VirtualHost>
Enable HTTP Strict Transport Security protocol in IIS To enable HSTS in IIS, do the following:
Add a Strict-Transport-Security header to the web.config file under the IIS installation root directory:
<system.webServer>
<httpProtocol>
<customHeaders>
<add name="Strict-Transport-Security" value="max-age=31536000"/>
</customHeaders>
</httpProtocol>
</system.webServer>
Restart IIS
Permissions
setfacl -R -m u:userid:rwx foldername
JAVA
rpm -qa | grep openjdk | xargs yum -y remove
GCP
sudo journalctl -u google-startup-scripts service
AWS
echo "find all the doggos, distract them with the yumz" > battleplans.txt
aws kms encrypt \
--key-id alias/catrobot \
--plaintext fileb://battleplans.txt \
--output text \
--query CiphertextBlob \
| base64 --decode > not_battleplans.enc
aws kms decrypt \
--ciphertext-blob fileb://not_battleplans.enc \
--output text \
--query Plaintext | base64 --decode > decryptedplans.txt
How to install MySQL on macOS
# On macOS, you can install MySQL easily using Homebrew Run.
brew install mysql
# You can now start the MySQL server by running:
brew services start mysql
# Now we need to secure the MySQL server. By default the server comes without a root password, so we need to make sure it’s protected. Run:
mysql_secure_installation
# The procedure can take a while, but it gives a lot of power to make sure you get the best defaults out of the box:
#Since we used brew services start mysql to start MySQL, your Mac will re-start it at reboot. You can run:
brew services stop mysql
# to stop this from happening, and also to immediately stop MySQL.You can also avoid this daemon mode (that’s what we call programs that always run in the background and restart when the computer is restarted) by running:
mysql.server start
#This will start MySQL and will keep it running until the computer is shut down, or until you run:
mysql.server stop
#and it will not re-start it at reboot.It’s up to you to decide which one you prefer.Now you can connect to the server using the command:
mysql -u root -p
#You will need to type the root user password after you run this command
Kubernetes
1. Using kubectl get all
# Using the kubectl get all command we can list down all the pods, services, statefulsets, etc. in a namespace but not all the resources are listed using this command. Hence, if you want to see the pods, services, and statefulsets in a particular namespace then you can use this command.
kubectl get all -n namespace
2. Using kubectl api-resources
# The kubectl api-resources enumerates the resource types available in your cluster. So we can use it by combining it with kubectl get to list every instance of every resource type in a Kubernetes namespace.
kubectl api-resources --verbs=list --namespaced -o name \
| xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>
# In the code above, provide your namespace in place of <namespace> and can run the above command. For too many resources present in a namespace, this command can take some time.We can use the above command, but a better variant of that would be something I found on Stackoverflow, where the above code has been converted into a function, which makes it more intuitive to use.
function kubectlgetall {
for i in $(kubectl api-resources --verbs=list --namespaced -o name | grep -v "events.events.k8s.io" | grep -v "events" | sort | uniq); do
echo "Resource:" $i
kubectl -n ${1} get --ignore-not-found ${i}
done
}
# All we have to do is provide the namespace while calling the above function. To use the above function, copy the complete code and paste it into the Linux terminal, and hit Enter.
# Then you can call the function:
kubectlgetall singlenode
# To list down all the resources in the singlenode namespace. This function will be available for use in the current session only, once you logout of the machine, this change will be lost and you will have to again define the function first and then use it in the next session.
3. Using kubectl get
# We can also use the simple kubectl get command to list down the resources we want to see in a namespace. Rather than running kubectl get command for each resource kind, we can run it for multiple resources in one go.
# For example, if you want to get pods, services, and deployments for a namespace, then you would run the following three commands:
kubectl get service -n singlenode
kubectl get pod -n singlenode
kubectl get deployment -n singlenode
#Well you can combine these three commands into a single command too,
kubectl get service, pod, deployment -n singlenode