I can Linux and So Can You (Bash commands) pt.2

So we can navigate and manage files from Bash. Now let's look at how to view, edit, and alter file permissions. So let's get started.

First, let's create a directory to work in. For that we can use the command mkdir. Let's also keep in the Documents folder.

$ cd Documents
$ mkdir testing
$ cd testing

So let's create a blank file. There are a few ways to do this. There is the touch command and simply writing nothing to a file name. The touch command is used for some extra stuff, so let's start with just putting nothing to a file.

$ : > test

If you ls to view the files, you should see one called test. The : is a minimal function that does nothing. As a result, you then send the output, or lack there of, to a file. Now making a blank file doesn't really do much, so let's write something to the file.

$ echo "This is a test" >> test

This appends the string "This is a test" without the quotes to the file test. The difference between > and >> is that > write to the file overwriting the original and >> will append to the file. To quickly view this file, we use the cat command.

$ cat test
This is a test

Okay, so if we want to overwrite the file and replace it with something, we use > and if we want to clear it all we use : > and rm if we want to just delete it. So how about we look at editing a file.

On Linux systems, and a handful of others, vi is usually included. In fewer cases, vim is included. If you have vim, I'd recommend that, but for now I'll just go over some basic vi commands. The vi editor is a visual editor that can be very powerful. The problem often is that it takes some getting used to as it's not like most programs people use today. Let's open a blank file and make a quick script.

$ vi hello.sh

It should dump you into the vi editor. To type into the editor, we need to enter insert mode. For this we press the letter i. Type in the following.

#!/bin/bash
echo "Hello, world!"

Now let's save and exit. First press esc so we can enter a command. After that we start the command with : and we want to save, which is w, and quit, which is q. So we can type :wq and press enter. Now we have a file we can actually run, almost. If you want to run it right now, you can type in this.

$ bash hello.sh
Hello, world!

Okay, that's great, but let's make it so we can execute the file directly. For that, we need to learn the chmod command, or change mode. When we talk about modes, we are refering to the permissions for read, write, and execute. In this case, we are interested in execute permissions. The permissions are also broken down into owner, group, and everyone. So let's look at how that works.

User permissions can be represented by either a letter, or a number 0-7. Here's a basic rundown.

Execute: x and 1
Write: w and 2
Read: r and 4

To apply multiple modes with numbers, we simply add what we want together, for example:

--- is 0
--x is 1
-w- is 2
-wx is 3
r-- is 4
r-x is 5
rw- is 6
rwx is 7

I often times find that when setting up new stuff I use numbers and when simply modifying a singular permission to do something, like execute, I use letters. Now when using numbers, we will often use three numbers for the owner (u), group (g), and everyone else's (o) permissions in that order. So say we want to open up a file so that anyone and everyone can see, run, and even edit the file, we do it like so.

$ chmod 777 hello.sh

I'd recommend against that because it's bad practice. If you need to get back to normal, chances are it was 644, so we just put in like so.

$ chmod 644 hello.sh

We want to add just execute permissions. To accomplish this we can figure out the number (755) or we can add the execute permissions. To add permissions with a letter we use the + and the letter. To remove a permission we use - and the letter. So let's make it executeable.

$ chmod +x hello.sh

Now we can run the file as need be. That is simply by running them like so.

$ ./hello.sh
Hello, world!

Notice the ./ in front, that indicates we are running the script in the folder we are already in. Now if we do an ls -l we get something that looks like this.

$ ls -l
total 8.0K
-rwxr-xr-x 1 root root 34 Mar 15 21:28 hello.sh
-rw-r--r-- 1 root root 15 Mar 15 20:52 test

Notice the -rwxr-xr-x, this shows us our permissions. Now hold up a moment, what if we don't want everyone and their dog being able to run the script? Okay, let's remove the executable permission for everyone. Everyone will be o. So we remove it like so.

$ chmod o-x hello.sh
$ ls -l
total 8.0K
-rwxr-xr-- 1 root root 34 Mar 15 21:28 hello.sh
-rw-r--r-- 1 root root 15 Mar 15 20:52 test

Okay, so now everyone can just simply read that file. We can do even more complicated stuff like applying multiple modes, removing multiple modes and doing so to multiple sets of permissions by separating with commas. At that point, I'd rather just do some simple addition and figure out the numbers. If you want information you can always enter in this command.

$ chmod --help

That will print out the help information. For now, we got the basics. So let's get back to file editing. We could open up a text editor, find it, delete, then replace it, or we could do it with a command. While when doing it with a command is a bit blind, it's easily scriptable. For this we use the sed command. As with most commands, you can get more information with the --help option. So let's look at checking this out.

$ sed --help
$ sed -i 's/world/universe/' hello.sh
$ cat hello.sh
#!/bin/bash
echo "Hello, universe!"

So, that was pretty nice, but how does it work? Well, the -i option is the in place flag. This lets it know we want to edit the file and not just print the output to standard output. The second part is what is called a Regular Expression, or regex for short. Regular expressions are an extremely complicated subject that has a lot of power to give at the cost of potentially massive headaches. But understanding the basics are a good thing to know. For now I will forego the description and stick to the basic examples. The real use here is allowing the creation of automated scripts to configure things for you as most configuration files are text based in the wonderful world of Linux.

So what can we do with sed? Using the regex, here's a few options.

s/this/that/ substitutes this with that.
s/this/that/g does the same thing, but for every instance of this.
/this/d deletes this.
s/this//g deletes all instances of this.
/something/s/this/that/ replaces this with that on line that matches something
y/abc/def/ transforms abc to def
/something/y/abc/def/ transforms abc to def on line that matches something
/this/p prints all the lines matching the pattern

With good regex experience, you can accurately automate editing of files how you need, of course with an understanding of possible expectations. With too much reliance, you could have a world of headaches. As a good practice, before editing a file, you may want to copy it to keep an old version just in case. In fact, it's something you'd do enough that it's actually built into the command! So let's look at that. Let's change universe back to world, but keep the other one as a copy with the suffix .old.

$ sed -i.old 's/universe/world/' hello.sh
$ ls
hello.sh  hello.sh.old  test

If you want, cat each file and see the difference for yourself. Or you could run each one. Either way, we got the basics down without getting into regex, so let's move on.

In more complicated situations, you will probably want to do this manually instead of spending hours on end creating a regex pattern to match something you can do yourself in a minute. Or perhaps we want to add more stuff to our script. Let's go back to the great vi editor. Let's open our hello.sh

$ vi hello.sh

So let's say we want to jump down to the very last line. For that, we type a capital G. The cursor should have jumped to the bottom of the screen. Now let's insert a line in here, so press i. Let's use some of the commands we've already done. Then add the following lines.

echo "This is a test" > testing
echo "This is appended to testing" >> testing
sed 's/test/something/' testing

After that, let's save and exit. Now let's run it.

$ ./hello.sh
Hello, world!
This is a something
This is appended to somethinging

Well, now isn't that weird? We don't want that somethinging. Okay, let's fix this, for that we need to make sure we only do this on the first line. So let's vi this file again.

$ vi hello.sh

If you've been following along, we need to edit line number 5. You can either arrow down to the line or enter the command :5 to jump down, remember to press enter. Now we can either arrow over to the spot we want to edit, on the ' or we can press w. So either arrow their, or type :5<enter>w. Now instead of pressing i, we press a. It works like this, i is to insert on the cursor, a is after, I is the beginning of the line and A is the end of the line. So press a, then type a 1 for line 1, then escape.

Okay, now let's say we want to save this as a different name, like newhello.sh. For the we type in :w newhello.sh and press enter. After that we can :q to get out of here and go run our new script. But wait, it says we can't! That's because although we saved the file as a different name, we still have the original open. After we alter a file, we need to explicitly save or ignore the changes to leave. Let's ignore it with the command :q! and press enter.

Okay, now we're out, and want to run our new file. If you try, you will probably get a permission denied error. That is because this new file did not inherit the permissions of the original. Let's copy those over, since we already know how to do it manually.

$ chmod --reference=hello.sh newhello.sh
$ ./newhello.sh
Hello, world!
This is a something
This is appended to testing

Woohoo! We're getting things done now. So now let's quickly run through some vi/vim basics that should be helpful in the future, or else we could be here a while if we keep stopping along the way.

$ vi newhello.sh

Okay, let's work on this file for good measure. As mentioned before, G jumps us down to the bottom, so do that.
Now let's jump to the top, we do that by pressing gg, yes that's twice tapping g.
To jump to the end of the line, we press $. To jump the the beginning of the line, we use 0. If there is whitespace at the beginning, we can get to the beginning of the line of text with ^.
Arrows can move the cursor around in the same directions, or you can do h for left, j for down, k for up, and l for right.
Let's jump to line 2 with :2<enter>.
Pressing w jumps you forward to the beginning of the next word or punctuation and a capital W jumps you based on whitespace. Likewise, b and B go backwards of w. The letter e jumps you to the end of a word and a capital E jumps you to the end before whitespace. So experiment with that on line 2.
Now, let us get back to the beginning of the line. Let's highlight the word test. To do that we want to jump to beginning four words down. So we type 4W<enter>. Now we press v to enter visual mode, then e to jump to the end of the word. From here we can do a lot of things, let's start with copying the word. Press y for yank.
Okay, it's copied, let's hit p to paste. Wait, what? ttestest? That's not right, is it? Well, p pastes right after cursor. Let's undo that with u. to redo, it's ctrl+r, not just r.
Let's try that again, press capital P. There we go, that pastes before the cursor. Now let's just delete this. So press these keys, bvex. The x key is the delete key that will delete directly under the cursor or whatever is highlighted in visual mode. So let's put test back, after all, we copied it. Press P. Hmm... that says testtest and not test. Here's the thing, unless we dump what we deleted into oblivion, it copies. If you don't want to copy it, before we press x, we do "_ to indicate a "black hole register." So instead it would have been bve"_x.
Okay, let's just replace it instead. So type in bvec, it will delete the word and dump us into insert mode, so now we can just type in test.
Press escape and now let's talk about what r does. If you type in r, it will replace the character directly under the cursor. Press r and then any character you want and you can see it replace. It's actually something I use quite a lot. Now if in the previous command we did r instead of c, and tried to type test, we would have gotten tttttttt as soon as we hit t. Might be useful when changing a hex code to black or white.
So let's say we want to copy the line. Tap y twice, yy. After that, regardless of where the cursor is, tap p or P. Now we have two copies of that line. To delete that line (cut the line). If you don't want to cut the line, you can do "_dd like with x.
On the topic of whole lines, there is a line visual mode. For that, it's a capital V instead of lower case v. Tap V to select all the lines you want, and like with insert, esc to exit if you need.
Finally, let's talk about copying to the clipboard. Sure, you can highlight the terminal and copy the normal way. The problem is, then you lose the powerful and quick navigation when in visual mode. We just need to copy to the correct register. Remember the _ black hole register? Well, we want the + register. So before using y, yy, d, dd, or x, type "+. Likewise we can do "+ before p or P to paste from the clipboard.

Okay, so quick rundown of registers, you can use a bunch of them. It's the character directly after the double quote ("). If you want to view registers or see where things are being copied to, run the command :reg and it will bring up a list of all the registers in use.

This is just a broad view of vi, but it will work for now. So we've edited files with sed and vi, we've altered permissions with chmod, we've sent output to files, so what else could we mess with? Access times for files are tracked. Often times we can use these with various ways to incrementally backup data. Because of this, there is a command to alter this. To do this, we touch the file. Let's start by looking at that wonderful help menu (:q! to leave vi, if you forgot already).

$ touch --help

So there are a few things I see touch frequently used for, the most common being making a blank file.

$ touch blankfile
$ : > blankfile

Both of these accomplish the same thing, just one take a little less effort. However, as mentioned, we can alter the access time. Here's the basics.

$ ls -l
total 20
-rwxr-xr-- 1 root root 144 Mar 16 13:06 hello.sh
-rwxr-xr-- 1 root root  37 Mar 16 12:30 hello.sh.old
-rwxr-xr-- 1 root root 145 Mar 16 13:21 newhello.sh
-rw-r--r-- 1 root root  15 Mar 15 20:52 test
-rw-r--r-- 1 root root  43 Mar 16 13:31 testing
$ touch test
$ ls -l
total 20
-rwxr-xr-- 1 root root 144 Mar 16 13:06 hello.sh
-rwxr-xr-- 1 root root  37 Mar 16 12:30 hello.sh.old
-rwxr-xr-- 1 root root 145 Mar 16 13:21 newhello.sh
-rw-r--r-- 1 root root  15 Mar 16 18:44 test
-rw-r--r-- 1 root root  43 Mar 16 13:31 testing

Look at that, we altered access time. So let's try something else.

$ touch -t 7001010000.00 test
$ ls -l
total 20K
-rwxr-xr-- 1 root root 144 Mar 16 13:06 hello.sh
-rwxr-xr-- 1 root root  37 Mar 16 12:30 hello.sh.old
-rwxr-xr-- 1 root root 145 Mar 16 13:21 newhello.sh
-rw-r--r-- 1 root root  15 Jan  1  1970 test
-rw-r--r-- 1 root root  43 Mar 16 13:31 testing

Look at that? Isn't it... epoch? Just wanted to make that joke. Okay, so you can do a handful of things with this. There's an -a option, which keeps it to altering just access time and not the modification time. With the -t we can do a 2 year or 4 year date. In the case of this example, we use a 2 year date for 1970, the 01 for month, 01 for the day, 00 for the hours, 00 for minutes and .00 for seconds. There is also the -d option that can take a date string, but the numbers are a little easier. The -m option alters just the modification time.

Another thing to mention is that the -c option to prevent creating a file if no file exists. If you may accidentally create an artifact because of a goof or something, might want to consider that as an option.

These are just some of the basics of viewing, editing, and altering files. There are plenty more, especially if you have selinux enabled. For now, this should get you going for a lot of Linux stuff. Exploring basics of regular expressions and some more uses of vi and vim will be a great asset for the future. There are certainly many other tools to learn, but now we can do many things necessary to actually get to using a Linux system.

I Can Linux and So Can You (Bash commands) pt.1

Often times I here people talking about wanted to learn Linux but have no clue where to start. I also noticed that those that do rely almost solely on a GUI. So let's say screw that GUI and try doing things on Bash. Open up a terminal and let's get started.

So let's start with some background. Bash, the Bourne Again Shell is based on the Bourne Shell (SH) and is commonly used by Linux and even now Mac. Other shells may be used like Dash (on Debian), Almquist Shell (Ash on Unix, also used on BSD and old Android versions), Tcsh (on BSD), Korn Shell (ksh, variants like mksh are on Android). Most of them are very similar and use similar conventions. I'm going to assume you're on Bash because that seems to be the most common Linux one.

The prompt should end in either a $, which is a standard user or # which is the root user. The prompts themselves may all look different and are customizable. I'll show commands after the prompt and their output. So let's get our bearings here. Let's check out what shell we are in, for that run the command:

$ echo $SHELL
/bin/bash

The $SHELL is a variable that says what shell you're in. Okay, so let's see what directory we are in. Chances are it may say on your prompt, but let's do a command.

$ pwd
/home/matt

You may see a tilde (~) in your prompt, that's the directory. It's shorthand for your home directory. So let's get an idea of the files and directories where we are. It's a very simple command:

$ ls
bash_history  Documents  Music     Public     Videos
Desktop       Downloads  Pictures  Templates

This gives us a basic list of the directory contents. Often times this is actually an alias to include colors based on the file type, like blue for directories. Mine is set as an alias for "ls --color=tty" and may be different for you. Now let's check out some of the flags you may use for ls to see some more information.

$ ls -a
.               Desktop        .msf4       Videos
..              Documents      Music       .viminfo
.armitage       Downloads      .oh-my-zsh  .vimrc
.armitage.prop  .gnupg         Pictures    .wine
.bash_history   .ICEauthority  .profile    .zcompdump-XXX-L-5.6.2
bash_history    .java          Public      .zcompdump-XXX-L-5.7.1
.bashrc         .local         .rnd        .zsh_history
.cache          .maltego       .set        .zshrc
.config         .mozilla       Templates

The -a flag shows hidden files, which are denoted by the beginning period "." You may also notice a single "." and a double ".." These are special, the "." refers to this directory and the ".." refers to the parent directory. Now we may want to look at even more information about our files, like permissions and ownership. For this, we use the -l option, which can also be used with the -a option if you want along the lines of ls -a -l or ls -al.

$ ls -l
total 32
-rw-r--r-- 1 matt matt    0 Jan  3 14:52 bash_history
drwxr-xr-x 2 matt matt 4096 Dec  6 14:17 Desktop
drwxr-xr-x 4 matt matt 4096 Feb 12 13:49 Documents
drwxr-xr-x 2 matt matt 4096 Feb 16 15:15 Downloads
drwxr-xr-x 2 matt matt 4096 Dec  6 14:17 Music
drwxr-xr-x 3 matt matt 4096 Feb 16 15:15 Pictures
drwxr-xr-x 2 matt matt 4096 Dec  6 14:17 Public
drwxr-xr-x 2 matt matt 4096 Dec  6 14:17 Templates
drwxr-xr-x 2 matt matt 4096 Dec  6 14:17 Videos

As you can see, there's a decent amount of stuff, so let's go column by column. The first column first letter is the file type, in this case d is directory and - is a normal file. The next three letters indicate read, write, and execute permissions for the user. An r means they can read, w can write, x can execute but a - in its place means the permission is off. The next three letters are for the group permissions and the last three are for everyone. The next column after that is the number of links, followed by owner, group, size, then a date of last modification and time, then the name of the file. There are still yet more permissions you can view that come into play with selinux. That is the -Z option. An example on a raspberry pi running CentOS I have looks something like this.

# ls -Z
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 anaconda-ks.cfg
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 epel-release-latest-7.noarch.rpm
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 http_fping.mod
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 http_fping.pp
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 http_fping.tt
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 README

The extra column is for selinux stuff. It can add another layer of security on your system and is a little outside of what I want to start with. So let's move on, shall we?

So ls can list directories anywhere, you just simply need to specify a path. But how do we know what path to use? Well, let's go over some Linux directory basics. The main folder is the root, which is /. This is not to be confused with /root, as that is the root user and / is the root to the file system. All of your files get attached under the root directory. Your user file will likely be in /home/<username>. So let's start by taking a glance at the root directory, so a simple ls / should do the trick.

$ ls /
bin   home            lib32       media  root  sys  vmlinuz
boot  initrd.img      lib64       mnt    run   tmp  vmlinuz.old
dev   initrd.img.old  libx32      opt    sbin  usr
etc   lib             lost+found  proc   srv   var

So what are all of these? The ones you should learn to familiarize yourself with are bin, which contains your binaries, or executable programs and sbin which has system binaries. The boot folder has your information for the system booting, like grub (grand unified boot loader). The dev folder contains device files like hard drives, attachable media, discs, ports, sockets, and so on. The configuration files for most stuff on a system level will be in etc. The home folder contains user folders, except for root, that's in root. When you plug in media, it will often be in media. The mnt folder is used for mounting, similar to media I guess. The opt folder contains optional programs which may also be in the usr folder, along with other binaries. The proc folder contains process information and system information. The tmp folder contains temporary files and if you think you lost something or downloaded something and can't find it, you may want to look in here. Other than that, I don't often use or look in the other folders or files myself. They have uses, I just have never needed to actively go into them.

We can probe around further to look into say... etc by doing ls /etc. So now we can look around the system, so let's talk about moving. Specifically, we need to change directory. The command for this is cd, and is the exact same command on Windows cmd. So let's change to root by running this command.

$ cd /

Chances are your prompt has changed replacing a ~ with a /. I would show you, but it varies system to system and can be configured. However now when we do an ls, it will list the / folder and not /home/<username>. So now we can move around, but how do we really jump around? There are some simple quick ways around.

cd ~ will move you to your home directory.
cd - will take you to the directory you were just in, so you can quickly toggle back and fourth.
cd .. will take you up one directory.

With those, you can begin exploring. I'd show some examples, but some that would eat a lot of space real quick. With cd and ls, you're on your way to not needing a file browser! So what other common things do we do with a file browser?

To copy a file, we have the cp command.
To move or rename a file we have the mv command.
To delete a file we have rm.
To delete a directory we can use rm or rmdir if it's empty.

Okay, so let's run through these commands with some context. If we want to copy a file called test1 and call it test2 we would do this.

$ cp test1 test2

Easy enough, and of course file paths can be used to copy to another directory, like so.

$cp test1 ../test2

This would copy it to the parent directory. So let's look at moving and renaming, since it's the same command.

$ mv test1 ../Documents/
$ mv test1 testing123
$ mv test1 /etc/samba/smb.conf

The first command moves test1 to the Documents folder, perhaps say from Downloads. The second one simply renames the file. You can also move and rename at the same time by just including the file name like the third example. So how about deleting a file? Simple.

$ rm test1

Now let's say we have a directory we need to delete.

$ rmdir uselessDirectory

If the directory is not empty, we need to delete everything first. Luckily we can do it all in one shot.

$ rm -R uselessDirectory

The R stands for recursive. It's just a fancy way of saying it deletes everything from the bottom up. Here's where things get a little weird, some systems alias rm to always prompt you about deleting every file. If you don't want a prompt, you use the -f option like

$ rm -R -f uselessDirectory

If it still prompts, it's an alias. We can look at getting around that later. For now, we know we can navigate, move, copy, rename, and delete stuff. So now we can pretty much shove aside a file browser, right? Well... we can still use it for a couple more things, but this seems long enough. Next I'll go over viewing, editing, and altering files.