Skip to content →

Amir Eslampanah Posts

Creating a Website Scraper using Gecko Driver (for Firefox) and Selenium (on Pop Os Linux)

First install python and pip:

sudo apt-get update
sudo apt-get install python3 python3-pip

Now lets setup a virtual environment

pip3 install virtualenv

You may get a warning about the path, add the path as follows:

sudo nano /etc/environment
sudo source /etc/environment

Now lets create the virtual environment and install selenium

mkdir -pv selenium-firefox/drivers
virtualenv .venv
source .venv/bin/activate
pip3 install selenium pandas

Now download and extract the latest gecko driver from https://github.com/mozilla/geckodriver/releases/

wget https://github.com/mozilla/geckodriver/releases/download/v0.29.1/geckodriver-v0.29.1-linux64.tar.gz
tar -xzf geckodriver-v0.29.1-linux64.tar.gz -C drivers/

Now lets create a sample script (a simple download-er):

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.keys import Keys
import time

firefoxOptions = Options()

#firefoxOptions.add_argument("-headless")
driver = webdriver.Firefox(executable_path="./drivers/geckodriver", options=firefoxOptions)

#Navigate to the login page
driver.get("https://some-page/my-account/")

time.sleep(5)

#Login
username = driver.find_element_by_id("username")
username.clear()
username.send_keys("usernamehere")

password = driver.find_element_by_id("password")
password.clear()
password.send_keys("passwordhere")

persistLogin = driver.find_element_by_id("rememberme")
persistLogin.click()

time.sleep(5)

driver.find_element_by_name("login").click()

time.sleep(5)

#Head to assets page
driver.get("downloadurlhere")

condition = True
while condition:
	# loop body here
	try: 
		downloadList = driver.find_elements_by_id('download-single-form')
		
		for download in downloadList:
			download.submit()
			time.sleep(25)

		driver.find_element_by_css_selector(".next[value='next']")
	except:
		break
# end of loop

driver.quit()

Save as browser.py and run as follows:

python3 browser.py

Note that you can use pandas to do data manipulation if necessary.

Leave a Comment

Configuration of Automatic Updates on Amazon Linux Version 1 & 2

The Amazon Linux AMI is a supported and maintained Linux image provided by Amazon Web Services for use on Amazon Elastic Compute Cloud (Amazon EC2). It is designed to provide a stable, secure, and high performance execution environment for applications running on Amazon EC2.

However, for whatever reason it is not enabled with automatic security updates on by default.

So lets fix that.

Step 0:

First lets install a version locking system for yum package manager. The reason we want something like this is to lock any and all packages that are too sensitive for automatic updates. A good example of this is any package where the version number can affect functionality in a major way such as gcc or Java.

sudo yum install yum-plugin-versionlock

You can then lock any package/set of packages with a statement like the following

sudo yum versionlock java*

To view a list of the current package locks, you can

sudo yum versionlock list

To remove all current package locks

sudo yum versionlock clear

Step 1:

Next lets configure automatic updates by first installing yum-cron

sudo yum install yum-cron

Lets modify the configuration to only enable security updates

sudo nano /etc/yum/yum-cron.conf

In the commands section set update_cmd to security and save-exit (Ctrl+o + Ctrl+x)

This will do something akin to

sudo yum update --security

Now lets enable it to start

sudo service yum-cron start

On AMI Version 1 also run:

sudo chkconfig yum-cron on

On AMI Version 2 also run:

sudo systemctl enable yum-cron

And you’re done!

Leave a Comment

Mounting S3 Buckets on Ubuntu/Pop Os Linux

First lets install the open source s3 Fuse

sudo apt-get install s3fs

Next you’ll need an access key id and secret access key; and you can get one from here

https://console.aws.amazon.com/iam/home?#/security_credentials

We’ll create a password file using these credentials

echo ACCESS_KEY:SECRET_ACCESS_KEY > ~/aws_s3.key
chmod 600 ~/aws_s3.key

The next step is to create the mounting point directory

mkdir ~/mount/s3-bucket

Now for the actual mount command

s3fs bucketname ~/mount/s3-bucket -o passwd_file=~/.aws_s3.key

Now you should be able to see the mounted files.

Leave a Comment

Installing Microsoft Teams on Ubuntu/Pop Os Linux

First lets add the repository key for the official release of Microsoft Teams for Linux

curl https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -

And lets add the repository

sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/ms-teams stable main" > /etc/apt/sources.list.d/teams.list'

Update and install teams

sudo apt install teams

Now you can login and use Microsoft teams!

However, you may encounter audio issues such as but not limited to:

  • No audio heard from call
  • Microphone cannot be heard in a call
  • Incorrect audio device is being used

These things can happen because for what ever reason Microsoft Teams for Linux is very bad at picking audio devices.

For this reason, I recommend installing PulseAudio Volume Control

sudo apt-get install pavucontrol

Then you can use the PulseAudio Volume Control to disable outputs and inputs you are not currently using, that way, teams can’t choose the wrong one.

Leave a Comment

IntelliJ Data Views & Java Collections

By default in IntelliJ IDE the internal view of a collection is hidden.

This is rather useless when debugging complex custom collections; so to avoid a bunch of extra debug code, you can simply disable the alternate view of collections in the customize data views option. (Ctrl+Shift+A)

You can also enable showing of static fields in here too which I recommend you turn on.

Now we can see the innards of the collection when debugging – YAY!

Leave a Comment

Setup OpenVPN Server with Streisand VPN on Ubuntu 16.04 LTS and connect with Windows Client

Let’s say that you need a run-of-the-mill VPN for your own purposes and you aren’t worried about using VPN-detecting services like Netflix then OpenVPN is a great option. Unfortunately it’s also a configuration nightmare; so lucky us then that the folks over at Streisand VPN have taken care of that.

Step 0 : Unattended Upgrades

The preparation step is to install unattended upgrades as an un-patched web-server is a really bad thing

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install unattended-upgrades apt-listchanges

Now to ensure /etc/apt/apt.conf.d/20auto-upgrades exists

dpkg-reconfigure -plow unattended-upgrades

Lets give it a test run to ensure things aren’t configured wrong

sudo unattended-upgrade -d

Step 1 : Setup of OpenVPN through Streisand VPN

Streisand is a VPN setup script that automatically installs and configures many popular VPNs including OpenVPN and WireGuard

First we need to generate an SSH keypair for Streisand to use (back these up since they will be necessary to login from SSH after a reboot)

ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Now we need to install the dependencies and clone the github (note we don’t use the official github address here since there is a bug-fix we would like)

sudo add-apt-repository ppa:ansible/ansible-2.8
sudo apt-get update
sudo apt-get install ansible python
git clone https://github.com/akumaburn/streisand.git && cd streisand

Now execute the setup script, and be sure to enter your server’s IP address when prompted.

Also do NOT USE the default install, you can use all the default values in the custom install EXCEPT do not install wireguard and shadowsocks.

./streisand

At a certain point you may get an error about some tunnelblick issue, simply hit Ctrl +C and then C to continue

Once it is finished you will have your documentation available to you in the ~/streisand/generated-docs/tunnel.html file

Save it somewhere and open the file in a web-browser and follow the instructions

Step 2 : Configuration of OpenVPN

By default, OpenVPN is configured by the installation script to not allow multiple clients to use the same common-name (mentioned in the certificate), if you wish to change that you can by

sudo nano /etc/openvpn/server.conf

Find and un-comment this line so that you won’t need to generate a unique certificate for every VPN client

duplicate-cn

And hit Ctrl+o and Ctrl+x to save and exit

Now simply restart the OpenVPN server

sudo systemctl restart openvpn@server

Now you need to download the OpenVPN Connect Application and import the config file (you can pick any from the list on your website after following the tunnel.html instructions)

Now you should be able to connect

That’s it 🙂

Leave a Comment

Windows Tweaks and Fixes

This is a collection of scripts I regularly use to fix windows problems; some of them I have written myself and some I’ve gotten from others.

ClearEventViewerLog.ps1 – Clears the event viewer log that can get easily bloated over time

DisableMemoryCompression.ps1 – Disables Windows Memory Compression, which should improve application latency at the cost of a little extra memory

EnableLongPaths.ps1 – Enables Windows to deal with file paths that are longer than 255 characters

Fix_Wifi_Ping_Spikes.bat – Enables/Disables Windows WiFi Discovery; when it is off ping spikes are dramatically reduced, sometimes from 70ms down to 2ms

Regular_Maintenance.bat – Very powerful windows repair script that I wrote, does the usual DISM and SFC stuff, and also fixes the Windows Store, Start Menu and other Windows exclusive packages – This script can often bring Windows 10 back from the brink of death

Bring Back/Remove Open With Option on Context Menu.bat – This script will restore the “Open With” context dialog in the event that it has gone missing

SearchAndPackagesFix.ps1 – This script I wrote attempts to re-install all windows packages and also tries to fix the Windows start menu (It is used by Regular_Maintenance.bat)

Decrapify.ps1 – A powerful third-party script that lets you remove bloatware from Windows (be very careful what you select to remove as some packages are required by Windows)

You can download them all as a zip file here -> https://amireslampanah.com/Tweak_Scripts.zip

The password is: “amire”

(without the quotes)

Additionally I like to use BleachBit to clean the temp file mess created by Windows update and Wise Registry Cleaner(This link gives you a discount you can’t ordinary get) to clean the Windows Registry.

Discount on Advanced System Care 10 with Driver Booster Pro enter code 25OFF

Leave a Comment

Installation & Configuration of GitBucket – A self-hosted alternative to GitHub

In the previous tutorial titled How to Install GitLab CE to have your own CI (Continuous Integration) with Source Control on Debian 10 we went over the advantages of GitLab. However GitLab is notoriously fickle in a production environment; updating it can easily break your entire workflow and it isn’t the most reliable solution for someone who just needs GUI-assisted commit tracking.

In comes GitBucket a fully self-contained GitHub clone (written in Scala) that you can host yourself and that runs off of the JVM.

Step 0:

First you will have to ensure that your machine has at-least Java8 installed.

In this case we will use the OpenJDK version (because we like open source and development) but note that you can also use the OpenJRE as well as the Oracle Java.

sudo apt-get install default-jdk

Check that your Java version is higher than 8

java --version 

In our case we have 11.0.9 which is good

openjdk 11.0.9 2020-10-20 OpenJDK Runtime Environment (build 11.0.9+11-post-Debian-1deb10u1) OpenJDK 64-Bit Server VM (build 11.0.9+11-post-Debian-1deb10u1, mixed mode, sharing)

Step 1:

Now lets download the latest gitbucket.war file from the releases page

In our case it is version 4.34

wget https://github.com/gitbucket/gitbucket/releases/download/4.34.0/gitbucket.war

And give it a test run using the terminal

java -jar gitbucket.war

Now it should show up when you go to yourwebsite.com:8080

Step 2:

Lets enable SSL via proxing from your main site

If you don’t know how to enable SSL in OpenLiteSpeed refer to my previous tutorial titled Installation and Configuration of OpenLiteSpeed with PHP, MariaDB, LetsEncrypt SSL, PHPMyAdmin, and NinjaFirewall on Debian 10 Buster

Create a new VHOST with whatever parameters you want

Now you’ll want to go to External App->+ sign to add an external app, choose type Web Server

Fill in the details

Now add a new context of type proxy by going to Context->+

Now go to SSL and fill in the SSL parameters from your main domain

Enable Rewrite to HTTPs

Create an A record in your nameserver configuration (in your nameserver’s management interface – typically your domain provider) to point the gitbucket.yourwebsite.com subdomain to your server’s IP address

Add it in both HTTP and HTTPs Listeners for your created VHOST

Save all changes and Do a graceful restart

Now you should be able to visit your subdomain gitbucket.yourwebsite.com and it should automatically redirect to the HTTPS link (this is vital since the connection needs to be secure for logins)

Step 3:

Now that its running we need to make it into a service so that it starts automatically on reboots

First lets move it to somewhere that makes sense for an executable

cd ~
mv gitbucket.war /usr/bin/gitbucket.war
sudo nano /etc/systemd/system/gitbucket.service

Paste the following contents and save

Description=GitBucket Server Service
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=always
RestartSec=1
User=root
ExecStart=/usr/bin/java -jar /usr/bin/gitbucket.war
[Install]
WantedBy=multi-user.target

Now you’ll want to run the following commands to update, start and enable it to run at boot

sudo systemctl daemon-reload
sudo systemctl start gitbucket
sudo systemctl enable gitbucket

Now it should survive between reboots

You’re done 😉

Leave a Comment

Fixing Automatic Screen Rotation in Ubuntu

With some laptops like the one in the Getting XIDU PhilBook Max to Play Nice with Linux tutorial there can be a different offset than the standard one that is expected.

In order to avoid staring at the screen at right angles (to the detriment of both your neck and sense of orientation) we must make some changes to the configuration file for the orientation matrix in the distro. In the previous tutorial we covered a situation where the distribution didn’t support automatic rotation out of the box; now we cover a situation where the distro does support it.

First lets fix the head cranking temporarily so we can keep our sanity as we do this

xrandr -o normal

Thanks to Kristian on Stack Overflow we know how to get the relevant model information

First lets get the driver name

sudo udevadm info -n /dev/iio:device0

Now lets get the vendor and product name

sudo dmidecode | grep Manufacturer
sudo dmidecode | grep Product

Now we need to replace the information into the brackets in the string below

sensor:modalias:acpi:[driver name]*:dmi:*:svn[Manufacturer]:pn[Product Name]:*
 ACCEL_MOUNT_MATRIX=0, 1, 0; 1, 0, 0; 0, 0, 1

In our case we end up with (using * as wildcards)

sensor:modalias:acpi:*KIOX010A*:dmi:*:svn*:pn*XN141A*:*
 ACCEL_MOUNT_MATRIX=0, 1, 0; 1, 0, 0; 0, 0, 1

Where ACCEL_MOUNT_MATRIX is the transformation matrix that represents our orientation

Now lets add it to the end of our list of display rotations, to do this we follow the instructions at the top of the /usr/lib/udev/hwdb.d/60-sensor.hwdb

sudo nano /etc/udev/hwdb.d/61-sensor.hwdb

Paste in the string you made above, and save

Now we need to get the system to adopt the changes

sudo systemd-hwdb update
sudo udevadm trigger -v -p DEVNAME=/dev/iio:device0
sudo service iio-sensor-proxy restart

Now the screen should rotate as expected!

Remember this was for my specific laptop and the transformation matrix necessary for your laptop may be different.

Leave a Comment

How to Install GitLab CE to have your own CI (Continuous Integration) with Source Control on Debian 10

Let’s face it,

In these crazy days of programs that span hundreds if not thousands of separate source files; keeping a track of just what changed where and when and which copy is actually the current one without any kind of source control can be a daunting task.

In comes source-control to the rescue, but it(GIT, SVN, Mercurial, etc..) isn’t very user friendly as is. Thus source control repositories with web-interfaces like BitBucket and GitHub have become very popular. These websites let you easily view changes in a graphically friendly environment as well as conduct merges, copies, forks, and various other handy source control operations with a click of a button.

The problem is that these services charge for the privilege of having a private repository, which protects your code from public viewing. That said, the bigger problem is that it is difficult to assure that your code isn’t being viewed from the service provider.

GitLab solves all these problems and adds CI functionality on top of it. Unfortunately GitLab is also quite the configuration monster when you attempt to use it without the bundled Ruby/Puma/Nginx, so we’ll have to proxy it instead of trying to run it directly through LSAPI (trust me I wasted a day going through the nightmare personally of trying to get it to work by manually installing the required ruby gems, setting the paths, etc.. it’s a hopeless cause)

Step 0:

You will need some way of running terminal commands on your machine, in this tutorial I use XSHELL but there are free alternatives such as putty.

First this tutorial assumes that you’ve already gone through the steps in my previous tutorial titled Installation and Configuration of OpenLiteSpeed with PHP, MariaDB, LetsEncrypt SSL, PHPMyAdmin, and NinjaFirewall on Debian 10 Buster

If you haven’t you can still follow the tutorial but note that there may be some divergence

Step 1:

We will need to download the package for our distribution of Linux. Luckily GitLab provides a repository installation script which handles a bunch of things for us, so lets grab that.

wget https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh

We need to set the permission for this script to be executable first, so

chmod +x script.deb.sh

Now to add the repositories simply run the script

sudo bash ./script.deb.sh

Step 2:

Now lets install GitLab CE (Community Edition)

sudo apt-get install gitlab-ce

Step 3:

You may have noticed the error about GitLab not being able to detect a valid hostname, so lets configure the external URL

Open /etc/gitlab/gitlab.rb with your text-editor of choice and make the following changes

First change

external_url 'http://gitlab.example.com'

To

external_url 'http://gitlab.yourwebsite.com:9191'

Optionally, you can change the default theme by UN-commenting this line

#gitlab_rails['gitlab_default_theme'] = 2

Now save and upload the file and run the following command to reconfigure GitLab

sudo gitlab-ctl reconfigure

First run of this command took my machine almost 5 minutes to complete so let it run.

Step 4(OpenLiteSpeed):

Now we need to configure the VHOST for this GitLab installation to coincide with the subdomain we picked earlier

Login to OpenLitspeed and navigate to the VHOST Configuration (If you don’t know where this is check my earlier tutorial).

Click add to add a new Virtual Host

Fill in the following:

Click Save

Note: You may get an error saying the vhost.conf file doesn’t exist; click to create it and click save

Now you should have GitLab in the vhost list

Click to edit it

Now head over to the general tab and modify the document root

Change it to

$VH_ROOT/html

Also run the following commands and upload a test index.html page to the html/ directory

mkdir /var/www/gitlab/html
chown lsadm:lsadm /var/www/gitlab/html

Next go to the Rewrite tab, Enable Rewrite and Auto Load from .htaccess

Next we need to point the subdomain to this vhost configuration before we configure SSL.

To do so go to the Listeners section and modify both HTTP and HTTPs Listeners

Create a new Virtual Host Mapping

Next if your primary website (yourwebsite.com) already has an SSL cert you can use that one (we will be modifying the certificate a bit later in order to add the subdomain). Otherwise you will have to generate a new cert (as is explained in my earlier tutorial).

Hit the refresh configuration button,

Next we need to adjust our DNS records accordingly to point to the new subdomain by adding an A record for that subdomain with the IP address of our server

Now we need to wait a while for the name-server records to update

Now we need to temporarily remove HSTS ( Strict Transport Security) on the main domain as it will force HTTPs but we haven’t extended our certificate yet

Its as simple as removing “includeSubDomains” and clearing your browser cache

After a while we should be able to visit gitlab.yourwebsite.com and see something

You’ll notice that the SSL won’t validate correctly if you don’t have a wildcard certificate

That means we will have to add the subdomain to the existing SSL certificate for it to be valid as well

Run the following command

certbot certonly --webroot -w /var/www/yourwebsite.com/html/ -d yourwebsite.com -d www.yourwebsite.com -w /var/www/gitlab/html/ -d gitlab.yourwebsite.com

Clear your browser’s cache, and refresh the OpenLiteSpeed configuration.

Now you can navigate back to the VHOST Configuration to force HTTPs

Ãdd the following Rewrite Rule

rewriteCond %{HTTPS} !on
rewriteCond %{HTTP:X-Forwarded-Proto} !https
rewriteRule ^(.*)$ https://%{SERVER_NAME}%{REQUEST_URI} [R,L]

Step 5:

We need to change some more settings in the GitLab.rb file to enable SSL

First, Disable LetsEncrypt (since we already generated a set of certificates for our webserver we don’t want GitLab trying to generate another)

Next, lets modify the External URL we setup at the beginning to point to HTTPS instead of HTTP

Change

external_url 'http://gitlab.yourwebsite.com:9191'

To

external_url 'https://gitlab.yourwebsite.com:9191'

Now, we need to symlink the existing certificates to the directory that GitLab expects them to be in

Your certificate and chain have been saved at: /etc/letsencrypt/live/website.com/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/website.com/privkey.pem

GitLab expects these to be in /etc/gitlab/trusted-certs instead and named in the following format

gitlab.yourwebsite.com.crt
gitlab.yourwebsite.com.key

Run the following commands

mkdir /etc/gitlab/trusted-certs/
ln -s /etc/letsencrypt/live/website.com/cert.pem /etc/gitlab/trusted-certs/gitlab.yourwebsite.com.crt
ln -s /etc/letsencrypt/live/website.com/privkey.pem /etc/gitlab/trusted-certs/gitlab.yourwebsite.com.key
chmod -R 755 /etc/gitlab/trusted-certs/
sudo gitlab-ctl reconfigure

If you visit https://gitlab.yourwebsite.com:9191 you should now see the GitLab Installation

However this port 9191 business isn’t ideal, so lets proxy this connection

Step 6:

First go to the GitLab VHOST we created earlier, and under External App, we’re going to add our Nginx webserver

Fill in the following settings (note that we are looping back through the external address)

Next, create a new context proxy

Fill in the proxy context definition

Save and Refresh the configuration and you’re done!

Important directories to remember for GitLab

/var/log/gitlab
/opt/gitlab
/etc/gitlab

Leave a Comment