Website Setup Scripts
This script was used to build
davidje13.com.
It will install various subdomain code (and necessary environments)
and install an nginx proxy server. It will also use the "let's encrypt"
service to get SSH keys for the site.
Security updates will be applied automatically. Other updates can be
applied by running:
sudo apt-get update;
sudo apt-get dist-upgrade -y;
sudo shutdown -r now; # if needed
sudo apt-get autoremove -y;
The subdomain code will be refreshed automatically each day if changes
are found.
Updates to this repository can be applied by re-running the
installer.sh script.
Deployment can be migrated to a new server (preserving data) using the ./migrate.sh script:
./migrate.sh \
~/.ssh/old-key old-user old-host \
~/.ssh/new-key admin new-host \
davidje13.com
AWS Instance
EC2
Use an ED25519 key pair
- for best security, generate a key locally:
ssh-keygen -t ed25519 -N '' -C "website" -f ~/.ssh/website.pem
- import the public key into AWS either via the UI
(Network & Security → Key Pairs → Actions → Import Key Pair, copy contents of
website.pub into box),
or via the CLI:
aws ec2 import-key-pair --key-name "website" --public-key-material "$(cat ~/.ssh/website.pub)"
Launch an instance with the following config:
- AMI:
ami-0306865c645d1899c (Debian 12, 64-bit, Intel)
- Architecture: Intel (note: MongoDB does not currently provide binaries for Debian on ARM)
- Instance Type: t3.micro
- Key pair: as created earlier
- Use a security group which allows "All traffic" outbound (IPv4 and IPv6), and inbound traffic on:
- 80 (public:
0.0.0.0/0 & ::/0)
- 443 (public:
0.0.0.0/0 & ::/0)
- 22 (your ip)
- Storage: 8GiB gp3
- Encryption enabled (KMS key can be left as default, using aws/ebs)
- IOPS: 3000
- Throughput: 125
- Instance auto-recovery: Default
- Termination protection: Enable
- Credit specification: Standard
{
"MaxCount": 1,
"MinCount": 1,
"ImageId": "ami-0306865c645d1899c",
"InstanceType": "t3.micro",
"KeyName": "website",
"DisableApiTermination": true,
"EbsOptimized": true,
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"Ebs": {
"Encrypted": true,
"DeleteOnTermination": true,
"Iops": 3000,
"SnapshotId": "snap-05b63c7cce6fcdf1e",
"VolumeSize": 8,
"VolumeType": "gp3",
"Throughput": 125
}
}
],
"NetworkInterfaces": [
{
"AssociatePublicIpAddress": true,
"DeviceIndex": 0,
"Groups": ["<fill in>"]
}
],
"CreditSpecification": {
"CpuCredits": "standard"
},
"PrivateDnsNameOptions": {
"HostnameType": "ip-name",
"EnableResourceNameDnsARecord": true,
"EnableResourceNameDnsAAAARecord": false
},
"MaintenanceOptions": {
"AutoRecovery": "default"
}
}
After creation:
- assign an elastic IP (this can be done after the install script completes waiting on DNS changes if migrating an existing deployment)
- assign an IPv6 address (see below, from step 5 if IPv6 already configured for VPC)
- Disable metadata (Actions → Instance settings → Modify instance metadata options)
Note: do not disable this when creating the instance, as it is required by AWS for configuring SSH keys.
- optionally purchase a savings plan to reduce costs in exchange for a minimum commitment:
CloudWatch
IPv6
To enable IPv6:
- Go to the default VPC configuration (or whichever VPC the instance is in):
- Select "Actions" → "Edit CIDRs"
- Select "Add new IPv6 CIDR" and choose one of the options
(davidje13.com uses "Amazon-provided")
- Go to the subnet(s) your EC2 instance(s) are in. For each one:
- Select "Actions" → "Edit IPv6 CIDRs"
- Select "Add IPv6 CIDR", and enter any available value for the block
(must be different for each subnet, and probably a good idea to start at
00 or 01)
- Save
- Still within the VPC section of AWS, go to the Route Table being used by the instance:
- Select "Actions" → "Edit Routes"
- Add the route destination:
::/0, target: "Internet Gateway"
(then select the same value already used for 0.0.0.0/0)
- Save
- Ensure the EC2 Security Group has been set up with
::/0 inbound rules, as described above.
Also ensure it has an outbound rule for all traffic to ::/0 (this will be auto-created by
the earlier steps, unless outbound rules have been customised)
- Go to the EC2 Instance
- Select "Actions" → "Networking" → "Manage IP addresses"
- Expand "eth0" and in the "IPv6" section choose "Assign new IP address"
- Either enter a specific IP address (e.g. for migrating an existing deployment) or leave
the field blank to assign a random IP address within the subnet range
- Select "Assign primary IPv6 IP"
(this will ensure the IP address does not change, allowing easier Route 53 config)
- Save
- Note the assigned IPv6 address and add
AAAA records to Route53 (see below)
Full guide from AWS: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-migrate-ipv6.html
Route53
(skip AAAA record if IPv6 has not been configured)
| type |
name |
value |
ttl |
| A |
<domain> |
<elastic ip> |
1 hour |
| AAAA |
<domain> |
<ipv6> |
1 hour |
| CAA |
<domain> |
0 issue "letsencrypt.org" |
7 days |
| CNAME |
*.<domain> |
<domain> |
7 days |
If the domain will not send email, the following should also be added:
| type |
name |
value |
ttl |
| TXT |
<domain> |
v=spf1 -all |
1 day |
| TXT |
*._domainkey.<domain> |
v=DKIM1;p= |
1 day |
| TXT |
_dmarc.<domain> |
v=DMARC1;p=reject;adkim=s;aspf=s |
1 day |
(source)
Installation
Once the EC2 & Route53 config is done, log in to the box:
ssh -i ~/.ssh/website admin@<public-address>
The key fingerprint can be verified from the EC2 dashboard;
- Select the instance
- Select Actions -> Monitor and troubleshoot -> Get system log (this shows the same logs as running
journalctl on the instance)
- Find the
BEGIN SSH HOST KEY FINGERPRINTS section
Once connected, run:
sudo apt-get install -y git
git clone https://github.com/davidje13/Website.git
Website/installer.sh
# this will probably ask for a restart, or will eventually complain about not having a domain
sudo shutdown -r now # SSH session will terminate - reconnect
cp Website/env/refacto.template.env Website/env/refacto.env
vi Website/env/refacto.env
# fill in any appropriate options then save
Website/installer.sh '<domain>'
The script will automatically wait for DNS records to be configured when
needed, so you do not have to set up the DNS before running the script.
Backup and restore
You can backup all state using the command:
Website/backup.sh # generates backup-*.tar.gz
The output file contains one backup file for each stateful service (by running
the backup.sh script of the service).
Restoring from a backup depends on the service. Unpack the global backup file,
and run the relevant restore script:
Refacto
# restore from backup-2020-10-03T12-00-00.tar.gz
Website/refacto/restore.sh backup-2020-10-03T12-00-00.tar.gz
There is also an option to restore data from a backup file during
installation.
Migrating
When migrating from an old server, downtime can be minimised by waiting until
the new server requests a backup file to load before shutting down and backing
up the old server:
- Begin deployment of new server (follow steps above) until prompted with:
"Enter filename to import refacto data from"
- On old server, run:
sudo systemctl stop refacto4080.service
sudo systemctl stop refacto4081.service
Website/refacto/backup.sh
- Copy the generated backup file to the new server
- Enter the filename in the prompt
- Wait for the script to prompt:
"Created background task waiting for (domain) DNS to point to this instance"
- Reassign the elastic IPv4, and update the AAAA DNS entry to point to the new
instance's IPv6.
Note: this will cause SSH sessions to end if they are using IPv4
- Installation should complete automatically.
Checking logs
Services store logs in:
/var/log/web-listener/*
/var/log/refacto/*
/var/log/nginx/*
To view all recent nginx error logs:
( cat /var/log/nginx/error.log /var/log/nginx/error.log.1; gunzip -c /var/log/nginx/error.log.*.gz ) \
| sort | cut -d' ' -f1,2,5- | less
To view all recent nginx access logs:
( cat /var/log/nginx/access.log /var/log/nginx/access.log.1; gunzip -c /var/log/nginx/access.log.*.gz ) \
| awk '{ print $4 " " $5 " " $1 " " substr($0, length($1 $2 $3 $4 $5) + 6) }' | sort | less
To view all recent Refacto logs:
( cat /var/log/refacto/*.log /var/log/refacto/*.log.1; gunzip -c /var/log/refacto/*.gz ) | sort | less
To view all recent Web-Listener logs:
( cat /var/log/web-listener/*.log /var/log/web-listener/*.log.1; gunzip -c /var/log/web-listener/*.gz ) | sort | less
To view current firewall stats (e.g. number of packets to particular ports):
sudo nft list table inet filter
To view packets sent to unknown ports:
journalctl -k --priority=0..warning --boot=all
Other debug logs also end up in the system journal, for example:
journalctl -u 'refacto*'
journalctl -u 'web-listener*'
journalctl -u stats-monitor.service
General system stats (CPU, RAM, Disk) are collected by a background daemon and stored in /var/www/monitor/logs.
These logs are flushed to disk rarely, but it is possible to trigger a flush of the latest data by running:
sudo systemctl reload stats-monitor.service
This will cause the data to be flushed at the next interval, i.e. within the next 10 seconds.
The data is stored in a binary format with an ASCII introduction, and old files can be gzipped to save space.
Post Setup
You should add the root domain to the
HSTS preload list