Wednesday, April 7, 2021

File Synchronization and Backup

 Having access to your files at any given time is a convenience some people today rely on so heavily, that as a person who often never really needs any particular files I find it humorous. Despite this, I have been rather obsessive about file access and backups in the past few years. This is so much so that I realized recently I have over 50 USB flash drives and 4 external hard drives accessible to the network. tow of the hard drives are specifically for TV and movies because I keep duplicates at the ready should an external fail on Kodi. One of them is an old Apple time machine that is ready to go, but I don't actually use it. Another is an old Seagate I have had for about 10 years or so now that keeps chugging along despite the slow speeds and for the small files I usually work with it works just as I need it. Taking it a step further, I have a RAID 1 setup on my main laptop with most of my documents duplicated onto a secondary laptop, and everything else was in Google Drive.

So here's the thing about all of this. Google Drive, like any other cloud service, can have terms changed, storage space altered, and on top of that, do you really want your files in "the cloud?" Truth be told you could easily just manually copy files onto each device you have and be relatively fine. So if you do have storage available, you should make use of it. So stop being lazy and back up your files yourself!

That being said, of course there are solutions that depending on the effort you put in, can be  Let me start out by saying that ANY local network solution can be an "anywhere" solution with tools like a VPN. I personally have OpenVPN setup at home because my Asus router has that option built in. You can create your own OpenVPN server fairly easily, but that's not the topic at hand for this.

For file synchronizing across multiple devices and types of devices, I was recently made aware of some rather cool software called Syncthing. The program name is rather unappealing, but the software is one of the coolest things I did not realize I needed in my life. What sets this apart from any solution I have seen before is that it is a "decentralized" file sync program. This means it uses a Peer-to-Peer model as opposed to the typical Server-Client model. If that doesn't make sense to you, view it like this. Normally you have a "centralized" server that handles all requests, files get uploaded to and downloaded from the server. In a peer-to-peer model, any device can serve files to any device asking and any device can ask any device that has the file for the file. This creates a huge advantage in file accessibility for a busy connection.

This means a few things to consider. The first is that you will need at least two devices to synchronize, not a hard thing to find now-a-days. The second is that rather than configuring a server, you must configure each device. As of writing this, I have four devices connected and the only limiting factor is a raspberry pi 4 connected to a very slow external. So here is some food for thought over any problems I have run into.

The first thing is be certain where you want the files stored and what you will sync. The reason being if you blanket a whole swath of folders, there is a chance some program specific or computer specific stuff will take up room on devices you do not need it to. I have separated all of my stuff to be synchronized out of the normal workflow and pieced it back together with shortcuts. I have also tried to break down the folders that get synced into separate items on the list for greater control and flexibility. The second thing to keep in mind is space on each device, what you need, and where. This is in addition to deciding a good folder structure.

With that in mind, there is a little catch on Linux. I have an external formatted in NTFS, which Linux is not exactly a fan of. I did this because the ntfs-3g stuff works better than ext drivers on Windows which love to lock up the file system and make everything unusable. If you have the same setup, the device should be mounted with the user you launch Syncthing under as the owner. If you launch it as root, there's probably nothing to worry about.

Head on over to the download page and install Syncthing on any and every device you want to be involved. For Windows, I suggest also installing the SyncTrayzor just for ease of access. To launch on Windows, you can use the SyncTrayzor in the start menu or you can launch it via command line with the syncthing command. To get to the console to set everything up you can either launch a web browser and go to or if you have SyncTrazor, there is an icon in your notification tray there is an icon that you can right click on and select "Open SyncTrayzor." We want to begin to lock things down a bit and make them better.

It will start out with asking you for statistical data, do as you want there. Then it will mention something about setting a username and password, let's do that. If you missed it on the box, just go to the "Actions" menu on the top right hand corner and select "Settings." Under the "GUI" tab at the top you can set a username, a password, and select a theme if you wish to change the color scheme. I also recommend you check the option to use HTTPS for the GUI because why wouldn't you encrypt when there's an option? Click save, then "X" out of the GUI and relaunch. If you are using the website, you need to change to if you checked the option. After that you should be all set where it has a default folder. Let's remove that. To remove a folder, you click on it and select "Edit" and then "Remove" on the edit page, bottom left corner this time. Now you can go through and add all the folders you want to sync, remember to organize them first. After you get all the folders set, you can start on the next device. Follow the same process. It does a good job at auto-discovery for when you add a device. When you add the device, the device you are trying to add will prompt for approval, nice little security measure.

When you have both devices set of any folder, if any, each is going to share (keep in mind if you want to merge a folder on both devices, sync one to the other and then move the files you want into that folder to make things easier). Just edit the folder on the device and use the "Sharing" tab up top and check any device you want to sync with.

Before you add more devices, I recommend you let everything sync and normalize before you add the next device. When you get a third setup, you will need to add it to  ALL devices you want to sync with, then on ALL the other devices elect to share with ALL devices expected to sync for the best stability and performance. You can keep a more-or-less centralized model, but I would take advantage of removing single point failures and have everything connect.

Now let's talk phone. Chances are the big thing is going to be syncing your photos to your computer. We can do that and ditch Google Photos or whatever other invasive "online photo album" software is the flavor of the day. It's generally more polite this way because chances are you will also help protect the privacy of friends and family that may make an appearance in your photos. Unless you're like me and the majority of your pictures are sleeping dogs, screw-ups at work, and memes.

That being said, just download the app for your phone. Add your folders in there, then we can add devices the cool way. Select the devices tab, hit the plus, then select that little QR code icon. Then on the device you need to add, on it's interface under "Actions" there is a "Show ID" option that gives you a QR code you can scan. As usual, it the device will want confirmation that you are trying to add it. Share your folders like normal.

So that's a quick overview of a fairly easy to use, decentralized, file sync program. So here are my thoughts.

The first is, it's really cool and I am quite happy with it. The interface is usable and it's just easy to setup and use. The only real critique I have is I really wish there was a built in function to move a folder. Instead I have to remove the folder in the control panel, move it on the system, then tell it to re-add the folder in the new location and wait for it to sync and a couple of times complain about "conflicting" files. Not a deal breaker for me, the advantages are just too good.

All that being said, I will continue in my search for secure and private alternatives for everyday conveniences and hope that you too will join me in taking the necessary steps to protect your data. Targeted marketing and advertisements on top of data breaches has hit levels it never should have all for the sake of what is convenient and I hope we can remedy it. If that's not your goal, then maybe you will at least find some cool software to check out.

No comments:

Post a Comment

Tag Cloud

.NET (2) A+ (5) ad ds (1) addon (4) Android (4) anonymous functions (1) application (9) arduino (1) artificial intelligence (1) backup (1) bash (6) camera (2) certifications (3) comptia (5) css (2) customize (11) encryption (3) error (13) exploit (5) ftp (1) funny (4) gadget (4) games (3) GUI (5) hardware (16) haskell (6) help (14) HTML (3) imaging (2) irc (1) it (1) java (2) javascript (13) jobs (1) Linux (19) lua (1) Mac (4) malware (1) math (6) msp (1) network (13) perl (2) php (3) plugin (2) powershell (8) privacy (2) programming (24) python (10) radio (2) regex (3) repair (2) security (16) sound (2) speakers (2) ssh (1) story (5) Techs from the Crypt (5) telnet (1) tools (13) troubleshooting (11) tutorial (9) Ubuntu (4) Unix (2) virtualization (2) web design (6) Windows (16) world of warcraft (1) wow (1) wx (1)