Showing posts with label bash. Show all posts
Showing posts with label bash. Show all posts

Monday, April 8, 2019

I Can Linux and So Can You (bash commands) pt.3

Previously we worked on viewing and editing files using cat, vi, and sed. Now let's talk about searching and reporting. While it would be simpler and easier to learn Python or Perl for these purposes, we are instead going to discuss grep, awk, and some basic regex. Despite the awkwardness of awk and the accessibility of regex for programming languages like Perl and Python, awk is still utilized by many. Other variations also exist such as gawk (gnu awk) and nawk (new awk), but awk works for both and generally is shipped with the box. So I will stick to that for now. First through, I think a discussion of regex is in order. So buckle up, because it's gonna get bumpy.

Sidebar, you can test these out with grep real quick. Simply do this:

$ echo 'The quick brown fox jumped over the lazy dog.' > test.txt

Then run the regex without the outside slashes like so:

$ grep --color -E 'regex here' test.txt
or
$ egrep --color 'regex here' test.txt

Regular expressions use lots of special syntax for searching and grouping complex text. Often times people will be familiar with some of it due to the commonality of certain methods. I'm sure most are familiar with "wildcards" (*). There are also some different types of regular expressions, but I will try to avoid specific things and keep this as generic as possible. So let's start with a sentence to search.

The quick brown fox jumped over the lazy dog.

Every letter in the alphabet there. Now let's assume we are trying to find this line. Let's work to match as much of this line as possible and cover as much as we can. To encapsulate all of it, I will keep it between two forward slashes because that's commonly how you will come across it. So let's talk about the start of the sentence. That can use a special character, in this case it's ^. There are many special characters ([\^$.|?*+(){}) which to match them litterall requires putting a backslash (\) in front of them. Next we want to match a capital letter. The thing we want is the nested character bracket ([) for this. This allows us to search a grouping of characters in a single position, for example vowels would be [aeiou], capitals would be [A-Z], numbers would be [0-9] or \d, lower case would be [a-z], alphanumeric would be [A-Za-z0-9] or \w. Okay, so let's make magic.

/^[A-Z]/

Okay, that will only match a string that begins with a capital letter. Now let's narrow it a little more. Next we want to match 2 non-space characters and a space. To match a non-space character we use \S and to match spaces (space and tab) is \s or a literal space is a space. Or we can do \w for a word character, or \l for a lower case. So all of these work.

/^[A-Z]\S\S\s/
/^[A-Z]\w\w /
Confused yet? Yeah, it's complicated with many ways to do the same thing, but they are slightly different. Okay, so now let's talk about the next word. It's five lowercase letters. We can do this pretty easy, there's a way to look for a pattern of n length or between n and m characters long. We do something like [a-z]{5} for five characters or something like say... [a-z]{2,5} would be between 2 and 5 characters long. So let's do the 5.

/^[A-Z]\S\w\s[a-z]{5} /

Okay, so now we have the word brown. Let's assume we don't know the length of the word we are matching, just that it's made of word characters. We know it's at least one character long. The plus (+) character comes into play here. A search plus a + will match 1 or more instances of it. So \w+ matches brown, \d+ matches 8675309 or 1.

/^[A-Z]\S\w\s[a-z]{5} \w+/

Can we get more complicated? Yes we can. We know the next word is fox and a space. Let's match literally anything after this of 0 or more characters. For that we use the period (.) which matches one of anything and asterisk (*) which matches 0 or more of the pattern. If you wanted to do one or more, you would do .+ to achieve that. We'll add a space after, so it would either match two spaces or a word followed by a space. The catch is that it will match from the end of brown to the last instance of a space.

/^[A-Z]\S\w\s[a-z]{5} \w+ .* /

Okay, so two things left, the word dog and a period. Let's say we want to match either dog, or cat or neither? Well, we can do one or both. To do or we use the pipe character (|), and to keep it clean we will use a group, which goes in parentheses (()). To match 1 or 0 instances of something, the pattern gets followed by a question mark (?).

/^[A-Z]\S\w\s[a-z]{5} \w+ .* (dog|cat)?/

Okay, the light at the end of this tunnel is near. Or is that a train? No matter, into the breach! The sentence ends in a period. We need to match a period. Two things. First, since the period is a special character we need to escape it with a backslash (\). Second, the end is marked with a dollar sign ($), like in vi and vim.

/^[A-Z]\S\w\s[a-z]{5} \w+ .* (dog|cat)?\.$/

Now look at that. What a mess that is. Obviously we won't often need super complicated stuff like that. There's still a lot more, but this should help get you started. For a better reference you can check out this regex quick reference, and it also goes over the difference in types of regex. If you want a good amount of flexibility, Perl compatible regex is usually the way to go.

So now let's talk commands. Searching and reporting are two things computers should excel at. Most Linux distributions today come with egrep and gawk and there are also counterparts like grep and awk, which they are based on. You can use the -E option with grep or egrep. I suggest using the color option if it's not already aliased on your distribution so you can see what part matches. I'll cover aliases in another portion. The awk and gawk commands are part of a type of programming language made for searching and reporting. Usually you can turn to Perl or another programming language, but awk works for a quick and dirty one-liner.

So let's start with searching. The grep and egrep commands can be used to search files or output of other commands. When you need to narrow your output to a readable level, this will be the go-to. I often times just use grep because I'm just searching for a word and not some expressions, but we'll check out both. I think a good populated folder to use for demonstration will be dev, there are consistencies in there. So let's take a look.

To get an idea of how many files are in dev, take a moment to just look.

$ ls /dev

I'm not going to post mine because that's a long list. Now let's say we are trying to find if a partition exists on our hard drive. Well, it's listed in here. We know our drive is sda, so how do we list all the sda drives? Well, we pipe the output from ls through grep to do a search.

$ ls /dev | grep 'sda'
sda
sda1
sda2
sda3
sda4
sda5
sda6

Cool, so here we see I have 6 partitions (The first three are Windows related, 4 is extended, 5 is kali, and 6 is swap). So let's assume we want to just make a cut-and-paste command to search for hard drives in other older systems. Some use hda instead of sda. Let's also assume we want to check for multiple hard drives, so there could be an sdb or an hdb. Easy.

$ ls /dev | grep -E '^(h|s)d[a-z]'

If we don't include the ^ at the beginning I get watchdog in the result for the hdo part. That's really all there is to it for simple searches of output. We can also search through the files of entire directories with the -r option.

Beyond search filters, it's often necessary to report the findings. Often times that reporting or searching will be useful to dump into an actual readable format. For now, let's use a custom file we will call searchtest.txt, here's what I put in it.

id:title:author
1:The Origins of Modern Science:Herbert Butterfield
2:Catch 22:Joseph Heller
3:1984:George Orwell
4:Animal Farm:George Orwell

I made a column title as well. So let's run a quick search.

$ grep -i 'george' searchtest.txt
3:1984:George Orwell
4:Animal Farm:George Orwell

The -i option tells it to ignore the case. Now this is all fine and dandy, but still a bit difficult to read. So let's clean up the reporting phase using awk. We'll discard the id and just print the title and author.

$ awk -v FS=: '{print $2,$3}' searchtest.txt
title author
The Origins of Modern Science Herbert Butterfield
Catch 22 Joseph Heller
1984 George Orwell
Animal Farm George Orwell

Okay, so we have a lot going on here. We are using -v to set some variables, in this case the Field Separator (FS) from the default of whitespace to colon (:). This allows us to access fields by a dollar sign ($) followed by the place number. In the case of this file, we have $1, $2, and $3. We then have an actual code block where we print fields $2 and $3 separated by the Output Field separator (OFS), which defaults to a space. Okay, this is still messy. We could alter the OFS to make it clearer. There is another method to format your print statements a little better, printf.

$ awk -v FS=: '{printf "%-29s|%s\n", $2, $3}' searchtest.txt
title                        |author
The Origins of Modern Science|Herbert Butterfield
Catch 22                     |Joseph Heller
1984                         |George Orwell
Animal Farm                  |George Orwell

Okay, so now we can actually read it. Here's how the printf is working. The %s is a string we're going to substitute in, in order of the arguments passed to printf. The %-29s is to make sure the string is padded to a length of 29 characters long and the - makes it align to the left, default without the - is to the right. Now let's say we want to not include that first line. We can add a filter for that.

$ awk -v FS=: '/^[0-9]/ {printf "%-29s|%s\n", $2, $3}' searchtest.txt
The Origins of Modern Science|Herbert Butterfield
Catch 22                     |Joseph Heller
1984                         |George Orwell
Animal Farm                  |George Orwell

So there we can see that it accepts a regex filter. You can use this to filter through, select, and print out reports of any text files you have. Learning more about printf will also all you to do a lot of formatting on the reporting as well. The awk and gawk commands happen to be programming languages in themselves as well, but going into all of that detail right now would be lengthy. So for now, let's discuss a bit more for the ins and outs of printf.

The printf function exists in many programming languages and even as a Bash command. You may be wondering what %s means. Well, the % indicates a control and the s indicates a string. In this case, it's printing a string. If you wanted to simply print a percent, you'd have to type %%. You also have %c, which prints a single character ascii decimal value, %d and %i which print numbers, %e and %E print a number in scientific notation, %f and %F for floating point numbers, %g and %G which print in either scientific notation or floating point (whichever takes fewer characters), %o prints numbers in octal, %u prints unsigned integers, and %x and %X print in hexadecimal where %X prints in uppercase and %x uses lower case.

Formatting modifiers can be added to it as well, like the - justifies to the left, default is to the right. A + tells it to print positive or negative signs. A number indicates the space it should use at minimum and a decimal, like 5.2 would indicate a 5 character width with a floating point precision of 2. A leading zero will pad a number with zeros instead of spaces. A # tells it to use an alternate form for certain numbers, like hexadecimal 0x prefix. A ' will print numbers in the thousands to use a comma separator. As you can see, most of the formatting revolves around numbers, but the main thing is setting a width lets you put your output in columns, which can make it easier to read.

So for example, if you did a printf with %07.2 to 12.3, you'd get

0012.30

It's seven characters wide, padded at the beginning with zeros and held to a precision of two.

For now that's enough, next I will cover applying the regex to searching and substituting with vi and sed.

Saturday, March 16, 2019

I can Linux and So Can You (Bash commands) pt.2

So we can navigate and manage files from Bash. Now let's look at how to view, edit, and alter file permissions. So let's get started.

First, let's create a directory to work in. For that we can use the command mkdir. Let's also keep in the Documents folder.

$ cd Documents
$ mkdir testing
$ cd testing

So let's create a blank file. There are a few ways to do this. There is the touch command and simply writing nothing to a file name. The touch command is used for some extra stuff, so let's start with just putting nothing to a file.

$ : > test

If you ls to view the files, you should see one called test. The : is a minimal function that does nothing. As a result, you then send the output, or lack there of, to a file. Now making a blank file doesn't really do much, so let's write something to the file.

$ echo "This is a test" >> test

This appends the string "This is a test" without the quotes to the file test. The difference between > and >> is that > write to the file overwriting the original and >> will append to the file. To quickly view this file, we use the cat command.

$ cat test
This is a test

Okay, so if we want to overwrite the file and replace it with something, we use > and if we want to clear it all we use : > and rm if we want to just delete it. So how about we look at editing a file.

On Linux systems, and a handful of others, vi is usually included. In fewer cases, vim is included. If you have vim, I'd recommend that, but for now I'll just go over some basic vi commands. The vi editor is a visual editor that can be very powerful. The problem often is that it takes some getting used to as it's not like most programs people use today. Let's open a blank file and make a quick script.

$ vi hello.sh

It should dump you into the vi editor. To type into the editor, we need to enter insert mode. For this we press the letter i. Type in the following.

#!/bin/bash
echo "Hello, world!"

Now let's save and exit. First press esc so we can enter a command. After that we start the command with : and we want to save, which is w, and quit, which is q. So we can type :wq and press enter. Now we have a file we can actually run, almost. If you want to run it right now, you can type in this.

$ bash hello.sh
Hello, world!

Okay, that's great, but let's make it so we can execute the file directly. For that, we need to learn the chmod command, or change mode. When we talk about modes, we are refering to the permissions for read, write, and execute. In this case, we are interested in execute permissions. The permissions are also broken down into owner, group, and everyone. So let's look at how that works.

User permissions can be represented by either a letter, or a number 0-7. Here's a basic rundown.

Execute: x and 1
Write: w and 2
Read: r and 4

To apply multiple modes with numbers, we simply add what we want together, for example:

--- is 0
--x is 1
-w- is 2
-wx is 3
r-- is 4
r-x is 5
rw- is 6
rwx is 7

I often times find that when setting up new stuff I use numbers and when simply modifying a singular permission to do something, like execute, I use letters. Now when using numbers, we will often use three numbers for the owner (u), group (g), and everyone else's (o) permissions in that order. So say we want to open up a file so that anyone and everyone can see, run, and even edit the file, we do it like so.

$ chmod 777 hello.sh

I'd recommend against that because it's bad practice. If you need to get back to normal, chances are it was 644, so we just put in like so.

$ chmod 644 hello.sh

We want to add just execute permissions. To accomplish this we can figure out the number (755) or we can add the execute permissions. To add permissions with a letter we use the + and the letter. To remove a permission we use - and the letter. So let's make it executeable.

$ chmod +x hello.sh

Now we can run the file as need be. That is simply by running them like so.

$ ./hello.sh
Hello, world!

Notice the ./ in front, that indicates we are running the script in the folder we are already in. Now if we do an ls -l we get something that looks like this.

$ ls -l
total 8.0K
-rwxr-xr-x 1 root root 34 Mar 15 21:28 hello.sh
-rw-r--r-- 1 root root 15 Mar 15 20:52 test

Notice the -rwxr-xr-x, this shows us our permissions. Now hold up a moment, what if we don't want everyone and their dog being able to run the script? Okay, let's remove the executable permission for everyone. Everyone will be o. So we remove it like so.

$ chmod o-x hello.sh
$ ls -l
total 8.0K
-rwxr-xr-- 1 root root 34 Mar 15 21:28 hello.sh
-rw-r--r-- 1 root root 15 Mar 15 20:52 test

Okay, so now everyone can just simply read that file. We can do even more complicated stuff like applying multiple modes, removing multiple modes and doing so to multiple sets of permissions by separating with commas. At that point, I'd rather just do some simple addition and figure out the numbers. If you want information you can always enter in this command.

$ chmod --help

That will print out the help information. For now, we got the basics. So let's get back to file editing. We could open up a text editor, find it, delete, then replace it, or we could do it with a command. While when doing it with a command is a bit blind, it's easily scriptable. For this we use the sed command. As with most commands, you can get more information with the --help option. So let's look at checking this out.

$ sed --help
$ sed -i 's/world/universe/' hello.sh
$ cat hello.sh
#!/bin/bash
echo "Hello, universe!"

So, that was pretty nice, but how does it work? Well, the -i option is the in place flag. This lets it know we want to edit the file and not just print the output to standard output. The second part is what is called a Regular Expression, or regex for short. Regular expressions are an extremely complicated subject that has a lot of power to give at the cost of potentially massive headaches. But understanding the basics are a good thing to know. For now I will forego the description and stick to the basic examples. The real use here is allowing the creation of automated scripts to configure things for you as most configuration files are text based in the wonderful world of Linux.

So what can we do with sed? Using the regex, here's a few options.

s/this/that/ substitutes this with that.
s/this/that/g does the same thing, but for every instance of this.
/this/d deletes this.
s/this//g deletes all instances of this.
/something/s/this/that/ replaces this with that on line that matches something
y/abc/def/ transforms abc to def
/something/y/abc/def/ transforms abc to def on line that matches something
/this/p prints all the lines matching the pattern

With good regex experience, you can accurately automate editing of files how you need, of course with an understanding of possible expectations. With too much reliance, you could have a world of headaches. As a good practice, before editing a file, you may want to copy it to keep an old version just in case. In fact, it's something you'd do enough that it's actually built into the command! So let's look at that. Let's change universe back to world, but keep the other one as a copy with the suffix .old.

$ sed -i.old 's/universe/world/' hello.sh
$ ls
hello.sh  hello.sh.old  test

If you want, cat each file and see the difference for yourself. Or you could run each one. Either way, we got the basics down without getting into regex, so let's move on.

In more complicated situations, you will probably want to do this manually instead of spending hours on end creating a regex pattern to match something you can do yourself in a minute. Or perhaps we want to add more stuff to our script. Let's go back to the great vi editor. Let's open our hello.sh

$ vi hello.sh

So let's say we want to jump down to the very last line. For that, we type a capital G. The cursor should have jumped to the bottom of the screen. Now let's insert a line in here, so press i. Let's use some of the commands we've already done. Then add the following lines.

echo "This is a test" > testing
echo "This is appended to testing" >> testing
sed 's/test/something/' testing

After that, let's save and exit. Now let's run it.

$ ./hello.sh
Hello, world!
This is a something
This is appended to somethinging

Well, now isn't that weird? We don't want that somethinging. Okay, let's fix this, for that we need to make sure we only do this on the first line. So let's vi this file again.

$ vi hello.sh

If you've been following along, we need to edit line number 5. You can either arrow down to the line or enter the command :5 to jump down, remember to press enter. Now we can either arrow over to the spot we want to edit, on the ' or we can press w. So either arrow their, or type :5<enter>w. Now instead of pressing i, we press a. It works like this, i is to insert on the cursor, a is after, I is the beginning of the line and A is the end of the line. So press a, then type a 1 for line 1, then escape.

Okay, now let's say we want to save this as a different name, like newhello.sh. For the we type in :w newhello.sh and press enter. After that we can :q to get out of here and go run our new script. But wait, it says we can't! That's because although we saved the file as a different name, we still have the original open. After we alter a file, we need to explicitly save or ignore the changes to leave. Let's ignore it with the command :q! and press enter.

Okay, now we're out, and want to run our new file. If you try, you will probably get a permission denied error. That is because this new file did not inherit the permissions of the original. Let's copy those over, since we already know how to do it manually.

$ chmod --reference=hello.sh newhello.sh
$ ./newhello.sh
Hello, world!
This is a something
This is appended to testing

Woohoo! We're getting things done now. So now let's quickly run through some vi/vim basics that should be helpful in the future, or else we could be here a while if we keep stopping along the way.

$ vi newhello.sh

Okay, let's work on this file for good measure. As mentioned before, G jumps us down to the bottom, so do that.
Now let's jump to the top, we do that by pressing gg, yes that's twice tapping g.
To jump to the end of the line, we press $. To jump the the beginning of the line, we use 0. If there is whitespace at the beginning, we can get to the beginning of the line of text with ^.
Arrows can move the cursor around in the same directions, or you can do h for left, j for down, k for up, and l for right.
Let's jump to line 2 with :2<enter>.
Pressing w jumps you forward to the beginning of the next word or punctuation and a capital W jumps you based on whitespace. Likewise, b and B go backwards of w. The letter e jumps you to the end of a word and a capital E jumps you to the end before whitespace. So experiment with that on line 2.
Now, let us get back to the beginning of the line. Let's highlight the word test. To do that we want to jump to beginning four words down. So we type 4W<enter>. Now we press v to enter visual mode, then e to jump to the end of the word. From here we can do a lot of things, let's start with copying the word. Press y for yank.
Okay, it's copied, let's hit p to paste. Wait, what? ttestest? That's not right, is it? Well, p pastes right after cursor. Let's undo that with u. to redo, it's ctrl+r, not just r.
Let's try that again, press capital P. There we go, that pastes before the cursor. Now let's just delete this. So press these keys, bvex. The x key is the delete key that will delete directly under the cursor or whatever is highlighted in visual mode. So let's put test back, after all, we copied it. Press P. Hmm... that says testtest and not test. Here's the thing, unless we dump what we deleted into oblivion, it copies. If you don't want to copy it, before we press x, we do "_ to indicate a "black hole register." So instead it would have been bve"_x.
Okay, let's just replace it instead. So type in bvec, it will delete the word and dump us into insert mode, so now we can just type in test.
Press escape and now let's talk about what r does. If you type in r, it will replace the character directly under the cursor. Press r and then any character you want and you can see it replace. It's actually something I use quite a lot. Now if in the previous command we did r instead of c, and tried to type test, we would have gotten tttttttt as soon as we hit t. Might be useful when changing a hex code to black or white.
So let's say we want to copy the line. Tap y twice, yy. After that, regardless of where the cursor is, tap p or P. Now we have two copies of that line. To delete that line (cut the line). If you don't want to cut the line, you can do "_dd like with x.
On the topic of whole lines, there is a line visual mode. For that, it's a capital V instead of lower case v. Tap V to select all the lines you want, and like with insert, esc to exit if you need.
Finally, let's talk about copying to the clipboard. Sure, you can highlight the terminal and copy the normal way. The problem is, then you lose the powerful and quick navigation when in visual mode. We just need to copy to the correct register. Remember the _ black hole register? Well, we want the + register. So before using y, yy, d, dd, or x, type "+. Likewise we can do "+ before p or P to paste from the clipboard.

Okay, so quick rundown of registers, you can use a bunch of them. It's the character directly after the double quote ("). If you want to view registers or see where things are being copied to, run the command :reg and it will bring up a list of all the registers in use.

This is just a broad view of vi, but it will work for now. So we've edited files with sed and vi, we've altered permissions with chmod, we've sent output to files, so what else could we mess with? Access times for files are tracked. Often times we can use these with various ways to incrementally backup data. Because of this, there is a command to alter this. To do this, we touch the file. Let's start by looking at that wonderful help menu (:q! to leave vi, if you forgot already).

$ touch --help

So there are a few things I see touch frequently used for, the most common being making a blank file.

$ touch blankfile
$ : > blankfile

Both of these accomplish the same thing, just one take a little less effort. However, as mentioned, we can alter the access time. Here's the basics.

$ ls -l
total 20
-rwxr-xr-- 1 root root 144 Mar 16 13:06 hello.sh
-rwxr-xr-- 1 root root  37 Mar 16 12:30 hello.sh.old
-rwxr-xr-- 1 root root 145 Mar 16 13:21 newhello.sh
-rw-r--r-- 1 root root  15 Mar 15 20:52 test
-rw-r--r-- 1 root root  43 Mar 16 13:31 testing
$ touch test
$ ls -l
total 20
-rwxr-xr-- 1 root root 144 Mar 16 13:06 hello.sh
-rwxr-xr-- 1 root root  37 Mar 16 12:30 hello.sh.old
-rwxr-xr-- 1 root root 145 Mar 16 13:21 newhello.sh
-rw-r--r-- 1 root root  15 Mar 16 18:44 test
-rw-r--r-- 1 root root  43 Mar 16 13:31 testing

Look at that, we altered access time. So let's try something else.

$ touch -t 7001010000.00 test
$ ls -l
total 20K
-rwxr-xr-- 1 root root 144 Mar 16 13:06 hello.sh
-rwxr-xr-- 1 root root  37 Mar 16 12:30 hello.sh.old
-rwxr-xr-- 1 root root 145 Mar 16 13:21 newhello.sh
-rw-r--r-- 1 root root  15 Jan  1  1970 test
-rw-r--r-- 1 root root  43 Mar 16 13:31 testing

Look at that? Isn't it... epoch? Just wanted to make that joke. Okay, so you can do a handful of things with this. There's an -a option, which keeps it to altering just access time and not the modification time. With the -t we can do a 2 year or 4 year date. In the case of this example, we use a 2 year date for 1970, the 01 for month, 01 for the day, 00 for the hours, 00 for minutes and .00 for seconds. There is also the -d option that can take a date string, but the numbers are a little easier. The -m option alters just the modification time.

Another thing to mention is that the -c option to prevent creating a file if no file exists. If you may accidentally create an artifact because of a goof or something, might want to consider that as an option.

These are just some of the basics of viewing, editing, and altering files. There are plenty more, especially if you have selinux enabled. For now, this should get you going for a lot of Linux stuff. Exploring basics of regular expressions and some more uses of vi and vim will be a great asset for the future. There are certainly many other tools to learn, but now we can do many things necessary to actually get to using a Linux system.

Thursday, March 14, 2019

I Can Linux and So Can You (Bash commands) pt.1

Often times I here people talking about wanted to learn Linux but have no clue where to start. I also noticed that those that do rely almost solely on a GUI. So let's say screw that GUI and try doing things on Bash. Open up a terminal and let's get started.

So let's start with some background. Bash, the Bourne Again Shell is based on the Bourne Shell (SH) and is commonly used by Linux and even now Mac. Other shells may be used like Dash (on Debian), Almquist Shell (Ash on Unix, also used on BSD and old Android versions), Tcsh (on BSD), Korn Shell (ksh, variants like mksh are on Android). Most of them are very similar and use similar conventions. I'm going to assume you're on Bash because that seems to be the most common Linux one.

The prompt should end in either a $, which is a standard user or # which is the root user. The prompts themselves may all look different and are customizable. I'll show commands after the prompt and their output. So let's get our bearings here. Let's check out what shell we are in, for that run the command:

$ echo $SHELL
/bin/bash

The $SHELL is a variable that says what shell you're in. Okay, so let's see what directory we are in. Chances are it may say on your prompt, but let's do a command.

$ pwd
/home/matt

You may see a tilde (~) in your prompt, that's the directory. It's shorthand for your home directory. So let's get an idea of the files and directories where we are. It's a very simple command:

$ ls
bash_history  Documents  Music     Public     Videos
Desktop       Downloads  Pictures  Templates

This gives us a basic list of the directory contents. Often times this is actually an alias to include colors based on the file type, like blue for directories. Mine is set as an alias for "ls --color=tty" and may be different for you. Now let's check out some of the flags you may use for ls to see some more information.

$ ls -a
.               Desktop        .msf4       Videos
..              Documents      Music       .viminfo
.armitage       Downloads      .oh-my-zsh  .vimrc
.armitage.prop  .gnupg         Pictures    .wine
.bash_history   .ICEauthority  .profile    .zcompdump-XXX-L-5.6.2
bash_history    .java          Public      .zcompdump-XXX-L-5.7.1
.bashrc         .local         .rnd        .zsh_history
.cache          .maltego       .set        .zshrc
.config         .mozilla       Templates

The -a flag shows hidden files, which are denoted by the beginning period "." You may also notice a single "." and a double ".." These are special, the "." refers to this directory and the ".." refers to the parent directory. Now we may want to look at even more information about our files, like permissions and ownership. For this, we use the -l option, which can also be used with the -a option if you want along the lines of ls -a -l or ls -al.

$ ls -l
total 32
-rw-r--r-- 1 matt matt    0 Jan  3 14:52 bash_history
drwxr-xr-x 2 matt matt 4096 Dec  6 14:17 Desktop
drwxr-xr-x 4 matt matt 4096 Feb 12 13:49 Documents
drwxr-xr-x 2 matt matt 4096 Feb 16 15:15 Downloads
drwxr-xr-x 2 matt matt 4096 Dec  6 14:17 Music
drwxr-xr-x 3 matt matt 4096 Feb 16 15:15 Pictures
drwxr-xr-x 2 matt matt 4096 Dec  6 14:17 Public
drwxr-xr-x 2 matt matt 4096 Dec  6 14:17 Templates
drwxr-xr-x 2 matt matt 4096 Dec  6 14:17 Videos


As you can see, there's a decent amount of stuff, so let's go column by column. The first column first letter is the file type, in this case d is directory and - is a normal file. The next three letters indicate read, write, and execute permissions for the user. An r means they can read, w can write, x can execute but a - in its place means the permission is off. The next three letters are for the group permissions and the last three are for everyone. The next column after that is the number of links, followed by owner, group, size, then a date of last modification and time, then the name of the file. There are still yet more permissions you can view that come into play with selinux. That is the -Z option. An example on a raspberry pi running CentOS I have looks something like this.

# ls -Z
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 anaconda-ks.cfg
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 epel-release-latest-7.noarch.rpm
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 http_fping.mod
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 http_fping.pp
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 http_fping.tt
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 README

The extra column is for selinux stuff. It can add another layer of security on your system and is a little outside of what I want to start with. So let's move on, shall we?

So ls can list directories anywhere, you just simply need to specify a path. But how do we know what path to use? Well, let's go over some Linux directory basics. The main folder is the root, which is /. This is not to be confused with /root, as that is the root user and / is the root to the file system. All of your files get attached under the root directory. Your user file will likely be in /home/<username>. So let's start by taking a glance at the root directory, so a simple ls / should do the trick.

$ ls /
bin   home            lib32       media  root  sys  vmlinuz
boot  initrd.img      lib64       mnt    run   tmp  vmlinuz.old
dev   initrd.img.old  libx32      opt    sbin  usr
etc   lib             lost+found  proc   srv   var

So what are all of these? The ones you should learn to familiarize yourself with are bin, which contains your binaries, or executable programs and sbin which has system binaries. The boot folder has your information for the system booting, like grub (grand unified boot loader). The dev folder contains device files like hard drives, attachable media, discs, ports, sockets, and so on. The configuration files for most stuff on a system level will be in etc. The home folder contains user folders, except for root, that's in root. When you plug in media, it will often be in media. The mnt folder is used for mounting, similar to media I guess. The opt folder contains optional programs which may also be in the usr folder, along with other binaries. The proc folder contains process information and system information. The tmp folder contains temporary files and if you think you lost something or downloaded something and can't find it, you may want to look in here. Other than that, I don't often use or look in the other folders or files myself. They have uses, I just have never needed to actively go into them.

We can probe around further to look into say... etc by doing ls /etc. So now we can look around the system, so let's talk about moving. Specifically, we need to change directory. The command for this is cd, and is the exact same command on Windows cmd. So let's change to root by running this command.

$ cd /

Chances are your prompt has changed replacing a ~ with a /. I would show you, but it varies system to system and can be configured. However now when we do an ls, it will list the / folder and not /home/<username>. So now we can move around, but how do we really jump around? There are some simple quick ways around.

cd ~ will move you to your home directory.
cd - will take you to the directory you were just in, so you can quickly toggle back and fourth.
cd .. will take you up one directory.

With those, you can begin exploring. I'd show some examples, but some that would eat a lot of space real quick. With cd and ls, you're on your way to not needing a file browser! So what other common things do we do with a file browser?

To copy a file, we have the cp command.
To move or rename a file we have the mv command.
To delete a file we have rm.
To delete a directory we can use rm or rmdir if it's empty.

Okay, so let's run through these commands with some context. If we want to copy a file called test1 and call it test2 we would do this.

$ cp test1 test2

Easy enough, and of course file paths can be used to copy to another directory, like so.

$cp test1 ../test2

This would copy it to the parent directory. So let's look at moving and renaming, since it's the same command.

$ mv test1 ../Documents/
$ mv test1 testing123
$ mv test1 /etc/samba/smb.conf

The first command moves test1 to the Documents folder, perhaps say from Downloads. The second one simply renames the file. You can also move and rename at the same time by just including the file name like the third example. So how about deleting a file? Simple.

$ rm test1

Now let's say we have a directory we need to delete.

$ rmdir uselessDirectory

If the directory is not empty, we need to delete everything first. Luckily we can do it all in one shot.

$ rm -R uselessDirectory

The R stands for recursive. It's just a fancy way of saying it deletes everything from the bottom up. Here's where things get a little weird, some systems alias rm to always prompt you about deleting every file. If you don't want a prompt, you use the -f option like

$ rm -R -f uselessDirectory

If it still prompts, it's an alias. We can look at getting around that later. For now, we know we can navigate, move, copy, rename, and delete stuff. So now we can pretty much shove aside a file browser, right? Well... we can still use it for a couple more things, but this seems long enough. Next I'll go over viewing, editing, and altering files.

Sunday, February 17, 2019

How to Linux or Something Like That

I am by no means a Linux expert, but at work I am the one they tend to defer to with "how do I Linux" questions. So maybe it would be worthwhile to make a small beginner guide on how to at least pretend you know what you're doing. So let's get out the basic "gotchas" of Linux and some semantics.

Linux is a kernel. Linux is to Ubuntu (or whatever you flavor is) as Windows NT is to Windows 7. I sometimes com across those snarky "Linux isn't an OS" comments online and it gets annoying, but this is the brief explanation to those.

Linux itself comes in different distributions referred to as "distros" or "flavors." If you're gonna talk about Mint, flavors seems like the better term, although it may get weird when you talk about Puppy being your favorite flavor. Sometimes different distributions are just the same one with a different desktop environment (DE) or software pre-loaded on it. Some desktops include things like Gnome, KDE, X11, xfce, MATE, LXQt, etc. There are a lot of desktop environments. To further add to some confusion, there are also window managers (WM) like xmonad, i3, and a million other ones. When discussing with others the environments they use, it can get complicated and layered because everyone uses a million and one things just to stick two windows on the screen at just the right location and spacing.

Personally, I use things mostly default because I often have to move around to a lot of different environments. I use CentOS minimal (no DE), Kali, Windows 7, Windows 10, Mac OSX Sierra, Ubuntu and various derivatives of, just to name a few.

If you're looking to pick the right distro for you, you can check out Distro Watch, they keep up to date on everything. Browsing there, you can find quite literally hundreds to choose from. Each one varies to some extent with the file structure, commands, or general way of doing things.

One of the big things for people starting out is the package manager. Without being able to install software, there are limitations for those that don't just make things themselves and package managers are the biggest help. It's one of the things that I loved when starting out with Linux. You can download or keep up-to-date all of your software in a simple manor. Each distro has its own way of doing things. Debian based has aptitude, RedHat based has yum, Arch based has pacman, and so on. Most have some form of package tool and special format.

I'm on Kali right now, so if I want to update all of my stuff, I may do something like:

apt-get update && apt-get upgrade -y

This is really dependent on what distro you're on, what kind of updates you're getting (like if you're on a rolling distro), and the like.

Now a brief look at file structures. The root file is /. It contains a handful of folders such as bin (user binaries), boot (boot files), dev (device files), etc (configuration files), home (user folders), lib (libraries), mnt (mount), opt (optional apps, usually ones you install outside of a package manager will end up here), proc (process information), root (root user, not to be confused with the file root), sbin (system binaries), srv (service data), tmp (temporary files), usr (user programs), and var (variable files such as logs, default web root for web services, etc.). Chances are you actually have many more than that, and it all depends on the distribution. I'm personally a fan of how Linux handles devices in the dev folder allowing you access to devices in similar ways you interact with files.

Shells. Your shell on a Linux system is an invaluable tool for when all else fails. Knowing how to use it is immensely helpful, and customizing it to make it your own is very satisfying. Most variations seem to default to Bash (Bourne Again Shell), based off of the Bourne Shell. There are numerous other variations such as Korn shell (ksh), Almquist Shell (ash), Dash (default for recent debian distros), C shell (csh), tcsh (unix compatible c shell that Mac used for a while), Friendly Interactive shell (fish), Z shell (zsh), and many more. Most of them have cross compatibility with minor changes here and there for syntax and the big distinction being how they handle interactive sessions.

Personally, I use zsh with Oh-my-zsh!, and tmux (a terminal multiplexer) when I'm getting down and dirty. For scripting, I stick to good ol' Bash... or Python or Tcl, depends on what I'm doing really. Tcl seems to pop up at random and I use it at work for a custom backup script for the many switches we have (with expect).

Some of the more basic things you would do to get started on Linux and really grab into the shell is basic file navigation and text editing. When it comes to text editing, I love Vim. My favorite thing about Vim/Vi is how quick and easy it is to do stuff when you know what you're doing. My second favorite things is when people copy a command with vi or vim in it and have no idea what they're doing. You see kids, Vi is a visual text editor that requires knowing commands in it to edit, save, and even exit it. There is also nano, which is to speed with most newcomers, but not every distro comes with it. So let's go over a quick and incomplete Vi/Vim lesson!

To quit the program you type :q
To move the cursor, you can use arrows or h (left), j (down), k (up), l (right)
To insert text (go into insertion mode), i inserts on the cursor, a inserts after, I inserts at the very beginning, A at the very end of the line
To stop inserting, press esc
Backspace doesn't always work, delete always does, x will delete when not in insertion mode
To save you type :w
To save and quit you can type :wq or :x
To quit without saving, you type :q!

There are also ways to copy and paste, regex substitute, find, and all sorts of things. If you start getting used to it now, you can soon start to learn all the other cool things.

Let's say we want to edit the hostname on a computer. We do something along the lines of:

cd /etc
vim hostname

Then press i, type in the hostname, press esc, then :wq
We can then restart the computer. So what are some quick and easy commands to get started?

cd <directory> (change directory)
mkdir <directory> (make directory)
touch <file> (create a file)
rm <file> (remove a file)
cp <file1> <file2> (copy a file)
mv <file1> <file2> (move or rename a file)

There are a lot of commands, and it changes from system to system to some degree. Without getting into one specifically, it's hard to go too in-depth.

So how do you pick a distro for you to get started? Major distributions are a good start with the large communities and lots of documentation and tutorials. Ubuntu, Mint, CentOS, Debian, Fedora, Manaro are just a handful of some of the major ones. I personally started out with Ubuntu. With Ubuntu I could do everything with GUI and introduce myself into command line stuff at my own pace. After that, I learned how to configure and run systems entirely from the command line.

Thursday, December 19, 2013

Purge old man pages for apropos

This is something I wanted to do for a while on Ubuntu, and I'm not sure if it will work on other distributions. The main problem is that I would remove a program and the man page for it would remain. This gets annoying when I'm searching for stuff and end up double checking on what was removed or decide to try a command I removed. The solution is simple, but takes a few minutes to run. As root, run these commands.

cd /usr/share/man
rm -R /var/cache/man
mandb

As simple as that and all the old stuff gets removed. This just simply removed the cached manual pages then reconstructs it.

Sunday, April 7, 2013

Shared Object with GHC 7

I was hitting my face on my desk for a few hours today trying to work this out. So here's my post to hopefully save anyone else who ended up stuck on this. I made a bash script to do it all, so here's the script. Note that this makes dynamic shared objects (.so files), still trying to figure out how to do it statically.

#!/bin/bash

TMPHELP="module_init.c"
FILENAME=${1%.*}
declare -i EXITCODE

cat <<TMPHELPER > $TMPHELP
#define CAT(a,b) XCAT(a,b)
#define XCAT(a,b) a ## b
#define STR(a) XSTR(a)
#define XSTR(a) #a

#include <HsFFI.h>

extern void CAT(__stginit_, MODULE)(void);

static void library_init(void) __attribute__((constructor));
static void library_init(void) {
    /* This seems to be a no-op, but it makes the GHCRTS envvar work. */
    static char *argv[] = { STR(MODULE) ".so", 0 }, **argv_ = argv;
    static int argc = 1;

    hs_init(&argc, &argv_);
    hs_add_root(CAT(__stginit_, MODULE));
}

static void library_exit(void) __attribute__((destructor));
static void library_exit(void) {
    hs_exit();
}
TMPHELPER

ghc -O2 --make -dynamic -shared -fPIC $2 -o "${FILENAME}.so" -optl-Wl,-rpath,/usr/lib/ghc/ -lHSrts_debug-ghc7.4.1 $1 $TMPHELP  >&1

EXITCODE=$?

set -x

if [[ $EXITCODE -eq 0 ]] ; then
    rm "${FILENAME}.hi"
    rm "${FILENAME}.o"
fi

if [[ -f ${FILENAME}_stub.h ]] ; then
    rm "${FILENAME}_stub.h"
fi

rm $TMPHELP
exit $EXITCODE

I'm honestly not 100% sure how everything works, but my main goal was to create shared objects so I could use them in Python. One thing to note if you are loading a shared object into Python made with Haskell, the IO functions don't seem to display on IDLE (works fine on the terminal).