Download a single page

Httrack is a great tool for download a whole website, or the internet if you don’t set up your options correctly. However sometimes we need to get just a single page and using the browsers save function just isn’t going to cut it.

So a bit of searching and here we go

httrack “http://www.google.com/” -O “./www.google.com” “http://www.google.com/” -v -s0  –depth=1 -n

This will download the entire page with javascript and styles and images so it works locally. Jolly good and nice and simple.

The only thing is it does put some of the assets in some deep directories but it isn’t to hard to work it out and for most situations will just work.

Updating NextCloud

I have been using NextCloud for a little while now as an alternative to Google drive, have been running it on my own server and things are going really well.

One thing that came up recently was that an update was required. NextCloud includes a handy little auto updater that whilst running into various issues did indeed work for the most part.

The first issue is a .user.ini file in the NextCloud install directory ended up being owned by root that meant NextCloud couldn’t write to it, that was an easy fix. I was unable to determine why that file got taken over by root, the webserver runs as Nginx so the system must have done something.

Well anyway after sorting that and continuing the update process it began creating a backup of files, at which point it timed out. However on refreshing the page it indicated the backup was done and it continued the download. This next part continued without any visual indication, just each time I tried to restart the download process I got a message that it was in process. Anyway that all finished without any further issues which was nice.

Next up reloading the NextCloud page gave me an update screen and a button, that didn’t work but I had read the manual update instructions: Link and saw that their was a command I could run to perform the next steps. The command for me was `sudo -u nginx /opt/php/bin/php occ upgrade` OK so that wasn’t the actual command, that isn’t my PHP directory but you get the idea. That command run and then everything worked as I expected it to.

There were a couple of other commands to run that would do various DB updates and also I now need to get imagick built for PHP 7.1 which should be fun. But all is now updated.

Brother MFC-J6510DW Network Printing From Antergos

So I have had this gigantic scanner / printer sitting on a desk unable to print for at least 6 months, and with a need to print onto bits of mushed up tree I needed to fix it.

In short the process was quite simple:

Download the Linux drivers from Brother, both the cups wrapper and LPR printer driver

I extracted the cups wrapper first, it goes into the /opt/brother/Printers folder so you end up with /opt/brother/Printers/mfcj6510dw/cupswrapper

In there I ran the brcupsconfpt1 file, this installed the drivers.

Then I also extracted the LPR drivers to the same location, this adds a inf and lpd folders into the /opt/brother/Printers/mfcj6510dw/ directory.

Then I had to go into Cups and configure the printer correctly http://localhost:631/ and once in there I actually found I had no rights to change anything. To solve this I had to add my user to the sys group sudo usermod -aG sys richard I discovered this was the case by reading the ever useful Arch wiki: https://wiki.archlinux.org/index.php/CUPS#Configuration specifically the configuration section it lists a file that shows the admin groups.

Once I was in cups I had to alter the printer entry (it added one when we ran the cups wrapper driver previously) and change it to use lpd to connect (lpd://BRW00225853EF12/BINARY_P1) and then select the driver a few screens later.

Now I do not feel I could easily replicate this by following my rough steps above, partly because there was a lot of trial and error in getting this working and various things I tried to get it to work. I am confident that this is the correct set of tasks that I completed to get it to work.

Antergos System Boot Failing After Updates

So I performed a full update of the system and then on reboot get thrown into emergency mode because the system is unable to mount /var along with several of my other data partitions. However once in emergency mode all partitions have been mounted fine. A curious one indeed, however after a bit of Googling I found the answer: https://stackoverflow.com/questions/23371594/systemd-udev-dependency-failure-when-auto-mounting-separate-partition-during-sta

It turned out that with a certain kernal parameter set it was trying to mount the partitions before it was ready to so it failed, this is a super simple over simplification

Was able to work around this, although it is a sort of a hack. Would still love to know why this is occurring in the first place, but it appears as though udev isn’t mounting mmcblk partitions until after systemd init is complete, which causes dependency errors if /etc/fstab calls for an mmcblk partition. udev checks the fstab, waits for an mmcblk device to be mounted and times out, THEN attempts to mount the device.

OK So as mentioned on that SO post is to change the kernel parameter, unfortunately I am running an already custom kernel to fix IOMMU PCI bindings for VGA pass through and the last thing I want to do is break that. Luckily the OP on that question posted a workaround that I ended up using. I am documenting it here so I can find it again easily:

1) Create systemd service to handle mounting the partition:

#/etc/systemd/system/mount-data-partition.service
[Unit]
Description=Mount Data Partition
DefaultDependencies=no

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/mount /dev/mmcblk0p4

2) Add a Wants dependency to this service within systemd-udev-trigger.service:

#/usr/lib/systemd/system/systemd-udev-trigger.service
[Unit]
Description=udev Coldplug all Devices
Documentation=man:udev(7) man:systemd-udevd.service(8)
DefaultDependencies=no
Wants=systemd-udevd.service mount-data-partition.service
After=systemd-udevd-kernel.socket systemd-udevd-control.socket
Before=sysinit.target
ConditionCapability=CAP_MKNOD

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/udevadm trigger --type=subsystems --action=add ; /usr/bin/udevadm trigger

This causes mount-data-partition.service to be called by and executed before systemd-udev-trigger.service. The mount command will then look for /dev/mmcblk0p4 in /etc/fstab, and mount as specified (in my situation, /var). Since /dev/mmcblk0p4 is now mounted, udev recognizes that the device exists and no longer times out while waiting for it. System continues to boot as normal.

Thanks to SO user schumacher574 for this, saved my bacon. Also for anyone who actually reads this I upvoted his answer on SO.

Pi Zero W – Set-up WiFi on command line (Raspbian Stretch Lite)

So I am starting a project using a raspberry pi and the very first step was to get it connected up and running with a WiFi connection. Luckily this is quite straight forward in Stretch and there are multiple ways to achieve this.

The method I used is to edit the /etc/wpa_supplicant/wpa_supplicant.conf file and add a network entry for each WiFi network you want to be able to connect to. The format is as follows:

network={
    ssid="<SSID IN HERE>"
    psk="<Your WiFi Password In here>"
}

You can add in as many network blocks as you need, then once done just a simple reboot and the WiFi will be connected.

Adding a country=GB line can also help with certain models and making sure the WiFi is using the correct frequencies.

There is another method to achieve this that involves creating a file on the boot partition of the SD card it will automatically be copied into the Linux file system and if the details are correct the WiFi will just work out of the box. I did not use this method on this occasion however and can’t exactly remember how to do it.

You can also use the following command to scan for available WiFi: sudo iwlist wlan0 scan.

Lastly if you want to be secure you can avoid writing your password in the network block via plain text by using the wpa_password utility like this: wpa_passphrase “testing” “testingPassword” where testing is the SSID and the second parameter is the password. This generates a string that you can use in the wpa_supplicant.conf file.

There are also various other options that are described here I didn’t want to go through all of them so only included the basics.

Zulucrypt

.. I miss Truecrypt.. Ok that may be a slight exaggeration but I did find it useful on occasion so hide sensitive information from prying eyes… Ok that is also an exaggeration, I have never had to hide anything from anyone but if I ever did then Truecrypt would have been awesome.

Sadly the devs decided to stop developing Truecrypt but luckily for us other alternatives arrived. One of my favourites (based on the “feature list”) is Zulucrypt. https://mhogomchungu.github.io/zuluCrypt/

So at least I can rest easy knowing that when I have something sensitive I need to keep secure I have a good open source option to do it.

I would not take this as a “review”, think of this as just a personal note for myself to make sure I don’t forget about this tool. I have not used it before.

Retroshare

Haven’t really used it but wanted to keep a note of it on here just to make sure I didn’t forget about it. This essentially lets you make your own social network that no one can spy on. The wiki article has a good overview: https://en.wikipedia.org/wiki/RetroShare

 

Backing up System

A while back I decided I needed to back up my system so using the Arch wiki put together this command to copy the whole HDD to a location excluding some directories. Works a treat and whilst not totally ideal this is what I am using for the moment.

rsync -aAXv –exclude=”/dev/*” –exclude=”/proc/*” –exclude=”/sys/*” –exclude=”/tmp/*” –exclude=”/run/*” –exclude=”/mnt/*” –exclude=”/media/*” –exclude=”/lost+found” –exclude=”/home” –exclude=”/var” –exclude=”/minecraft” –exclude=”/docker” / /home/*****/Backup

To VM or not to VM…

Ok so I run Linux on my main system and all is great. However I enjoy playing games on my computer and despite that the Linux offering for games is getting much better all of the time there are many games that are Windows only. So a couple of years ago I read about a way to solve this dilemma. Run Windows in a VM and use VGA Passthrough to give the VM full control of the graphics card.

Well after months of faffing about I got a Windows VM set-up and then a few months after I got a Linux one set-up. I won’t cover the full steps here, ~I used the following tutorial which worked really well for me (using Nvidia graphics card) https://bufferoverflow.io/gpu-passthrough/ I did discover a couple of other little “gotchas” that I wanted to record here so that I don’t lose them:

For the Windows VM

You need to install the VirtIO drivers within Windows to make sure you get full performance out of hardware. Redhat make these drivers available for download here.

Something else I had to do to get good sound audio was install an old AC97 audio driver and after that crystal clear sound quality. Getting this done was a bit of a nightmare but I did answer a question on Stack Overflow about this issue and those instructions still stand Crystal Clear Audio in Windows. The original YouTube video I found that showed me how to do it has disappeared but at least I wrote out those instructions 🙂

For Windows that was pretty much it, make sure to install the latest graphics card drivers and if you get stuck look on forums.

For the Linux VM

This one was a bit more tricky and required some more steps, I do need to point out that these steps were relevant the first time I installed the Linux VM (Mint 17.3 Mate) but my most recent time (with mine 18.1 Mate) I didn’t have to faff about with the nomodeset stuff or anything on the EFI command line. The very last step about changing the name of grubx64.efi still applies though. This is not so bad as you can manually navigate to that file in the EFI “settings screen” (press F12 on a reboot and in those options you can select the hard drive and the boot file) :

• Set nomodeset as kernel parameter
• Run Live CD and Install (antergos not work, Mint did)
• in Live CD chroot into new install and add nomodeset param to grub default file then update-grub – http://askubuntu.com/questions/145241/how-do-i-run-update-grub-from-a-livecd
• Then load up VM without live CD and wait for EFI Shell to kick in
• Change directory to FS0:\ and cd into fs0:\efi\ubuntu\grubx64.efi and run the file – http://askubuntu.com/questions/566315/virtualbox-boots-only-in-uefi-interactive-shell
• Once In – Mount the boot partition and rename the grubx64.efi to /EFI/boot/bootx64.efi (or whatever the efi boot needs) this is in a comment on that page.

Well that is it, this was one monster of a project to get working but I am so glad I have. I now have Linux as my main system and if I ever need it Windows is a command away. I do have to have 2 mine and keyboard plugged in just to make sure I can still access the host system if things go wrong but that is a small price to pay.

Thought I would add I use a GTX-970 and whilst haven’t benchmarked to discover how good the VM compares to bare metal but the performance is bloody brilliant.

What Ports??

Oftentimes when debugging some form of online service on Linux if can be handy to know what ports are currently open on the system you are working on. Back “in the day” we got this functionlity via netstat. Unfortunately that has now been deprecated so how do we achieve this.

In short ss

ss -lntu

Will generate similar output to the netstat command and with the -p command you can see the process id that has bound to the port as well.

I recommend using man ss for information on what all the command switches do.