Republishing: templating with integrity checking, {con}templation in accord with integrity is in accord with excellence, Aristotle meet virtual machines running virtual machines running containerized images of machines. Going to have to change the site tag to projects, programming and puns.
Why SSD drives just to USB boot? A little bit faster and a lot more stable.
Why a more stable raspberry pi?
This is a large search space.. Running a node for Web 3.0 services, a node for any decentralized infrastructure, DeFi, IPFS, the Graph, ETH[…]
After following several tutorials which failed along with thousands of line of code dedicated to automating the setup for USB boot from a sata drive [transferring or cloning from an SD card] it felt like duty to write a quick post. I hope this saves someone time, perhaps frustration, and certainly as in my own case, feeling silly for having not started with this obvious approach.
After all the ‘USB boot on Ubuntu 21.4 server with a raspberry pi 4’– + similar, tutorials and scripts all failed, I thought about it and… Eureka! I plugged it in. I just flashed the image directly to my drive via USB. After trying lots of cables I found that using one of the USB 180° male – male adapters to connect to the drive worked with most other combinations.
Then just flash it like you would an sd card, configure system-boot just like an sd card and boot with only the usb connection to the drive [no sd card].
I haven’t tested with other images or pi’s [only pi 4 64bit ubuntu 21.04 image for raspi]– as this is ancillary to my task at hand, but I certainly would try this before what seem like incredibly silly methods in retrospect from tutorials I came across.
Not a lot more work, but some. Here flash ssd drive and sd card and connect or insert them both. Then ssh to the device and do a full update and upgrade. If you want to really dig in see the documentation at:
Or just sudo apt install raspi-config and then enter the interface sudo raspi-config and you’ll find the bootloader options under ‘advanced’ sub menu. Select boot usb first and reboot into your ssd drive as root system. —
Programming 3D environments without many high level libraries makes clear why the visual-cortex is so large! [expensive it is to compute graphics?] Below are some excerpts of the low level of objects written to build from the ground up, everything. From vertex to triples, to triangles, and beyond– not recommended for anything besides painful learning. Especially with the verbosity of Java, here’s part of a triple class.
public class Triple
public double x, y, z;
public Triple( double a, double b, double c ) {
x = a; y = b; z = c;
}
public Triple( Scanner input ) {
x = input.nextDouble();
y = input.nextDouble();
z = input.nextDouble();
input.nextLine();
}
public void sendData( FloatBuffer buff ) {
buff.put( (float) x );
buff.put( (float) y );
buff.put( (float) z );
}
Finally built the castle in the sky
Which is then a part of a single vertex, like below; then with three vertices of 2 triple’s…. *drum roll* we can create a single shaded triangle.
public class Vertex {
private Triple position;
private Triple color;
public Vertex( Triple p, Triple c ) {
position = p;
color = c;
}
public Vertex( Scanner input ) {
position = new Triple( input );
color = new Triple( input );
}
public void sendData( FloatBuffer pb, FloatBuffer cb ) {
position.sendData( pb );
color.sendData( cb );
}
}
What is a world? What does it mean to have a world space and a horizon, Res Extensa. It turns out its not really that complicated or difficult to simulate a world, depending on resolution and complexity it can be computationally extensive.
Intimating the major elements involved in rendering a world and a 3-dimensional view point from within that space.
2D
How about simulating many individual agents, each acting according to their own internal rules based on the limited knowledge they have from their perspective [keeping this a bit simpler than game theory with humans]. Two dimensions is plenty, below demonstrates some interesting, beautiful behavior emerging from just 3 simple rules.
This is a python variation I wrote of Craig Reynolds, ‘boids’ algorithm from 1986. The behavior is determined by just 3 rules which each of the boids, or particles in this case, follows for itself. I ran this on my iPad and the compute slowed it to barely a crawl, I sped up the screen recording for this high action production.
There is actually one more rule here, a 4th force helping to keep them on the screen
separation: steer to avoid crowding local flock
alignment: steer towards the average heading of local flock
cohesion: steer to move towards the average position (center of mass) of local flock
Some simple programs like these [even simpler], are turing complete and some sets of rules even create seemingly never ending complexity. If that sounds totally insane then you need to visit Stephen Wolfram’s work.
This is a short little clip from recording the p5js program I uploaded onto a page of my blog on the left side. It allows for the variables controlling length of joining and acceleration/deceleration to be adjusted.
The following ones are also boids variants, these inspired by some of Daniel Shiffman’s work in his book on simulation nature . It has two flocks with different ranges of starting characteristics, also interactively, tapping the screen adds to each flock. A long screen press toggles what I call visualized entropic mode, were the history of the biods movements is retained. Then shaking the device will clear the screen. I built an APK for Nethunter and one for the Graphene android I had been been working on.
Clearly they behave in some very interesting ways despite being incredibly simple.
Another beautiful visual, the data for this was passively collected from all wireless signals within range.
The local neighborhood, in terms of wireless access points and clients, explored in 3D.
In order for language to stay active and alive to itself their must still be an audience for Philosophy. Whether the possibility of philosophy could be lost, if left in a state of wonder then one finds oneself where all philosophy begins.
Relevance of scalable intelligence to Knowing Oneself and the possibility of scaling minds in general. The possible appearance of singularity, and becoming.
The race of temporal minds with scaling intelligence. Novel appearance of the eternal in such a way that currently non-scaling minds could potentially begin to scale and become scaling.
My team and I wrote dork in python using scrum implementation of agile method, delivering working fully tested iterations of the code according to product owner specifications derived from customer requests in two week sprints. Documentation is on ‘readthedocs’. https://gitnick-dork.readthedocs.io/en/latest/readme.html
We used github for version control with travis-ci and sonarcloud for continuous integration and inspection. The full repository is public on github.
Each new game randomly generates a maze and factories populate the world with descriptions, interactive items and NPC’s from databases. Games can be saved and loaded with persistence of the game state, map, inventory, NPC’s and points.
The project turned out well and was a pleasure to work on. Working with team brought new difficulties the work and complexity of the project we were able to achieve within a short time frame was far beyond what could have been done individually. I have some future expansions planned for the game so stay tuned!
My Raspberry cluster has come a long ways since the first iteration I showed with a docker cluster running OpenFaaS. I added some of the new Pi4 with 8gig of ram, added an ssd drive which I networked shared to the entire cluster. I am just about to tear it down again for improvements so its time for a snap shot.
When rebuilding the cluster I wrote an Ansible playbook to bootstrap it so that I didn’t have to repeat the same 15 processes 10 times. There’s a few major repositories doing similar things but none of them matched my vision so I wrote my own. This repo is already deprecated from version changes to Ansible but I’ll update when I finished with the hardware improvements. The repository is here:
Besides the upcoming rework; if interested, note that inventory file needs to be changed to match Rpi IP's and those IPs should be static. This script doesn't set the static IP because I have a managed switch and am running them all on a virtual lan. Also unless you are my neighbor the time/zone task within the configurations or the universal tasks may also need a change.
Ansible: beautiful and simple
The Read.md from the repository:
To highlight a few, the playbook sets new password, transfers ssh keys, hardens the system, as well as getting the Kubernetes cluster up and going with one Master and transfers the config locally to make use of kubectl to control the cluster. Among the rest.
I adapted a project from Alex Ellis’ to light an led on the blinkt module for each pod being run on a node, giving an indicator of load, balance or any failures.
Bigger, faster, taller, stronger- and not just the hardware. The real gems have been automating the bootstrap with Ansible, managing with Kubernetes (especially lens) and deploying with Helm.
ssd to one of the rPi’s to have some larger persistent storage
I’ve been running a few services on the cluster, I added a SATA adapter and an SSD drive to one of the pi’s to host my own cloud storage. I installed Falco(cloud-native runtime security) through helm along with metalb for the load-balancer, an nginx ingress, with a let’s Encrypt deployment for certificates. Also used pi hole running on the cluster as my networks DNS, providing local caching and secure dns and dnssec for everything else.
The number’s aren’t too drastic here but this is all being caught despite my browsers running ublock-origin.
I forwarded from a domain name that I own to the services I was exposing to the internet so I could reach them whenever wherever. Exposing your own services on the net has a long list of items to go through in terms of safety but just to touch on the basics, long high entropy password which cannot be guessed within the lifetime of the universe is a good start.
Following this up with a dual factor hardware key, also not a bad idea.
This doesn’t address any of the network and firewall changes needed but hopefully I can get into a bit more detail when I have time to share about hosting my own cloud security and surveillance system.
Much more to come but here are some pictures showing a bit of how nice it is to run your own Nextcloud server with the universal support their community has created I have it running on Android, iOS, MacOS, and Ubuntu.
Then to give a look at lens and how slick it is for any who haven’t seen it, it is amazing and easily lets you move from controlling a few things through the CLI to easily managing a massive scaling design.
Going to try and catch up on this back log of things I’ve been working on! I think some android projects and Apps I’ve done and basic AI simulations may be next. Since I mentioned it here’s part of the hardware used to transition to fully manage my security systems with cloud access, no monthly fees and control of my own data, getting full throughput with deep packet inspection and ect,
This is an implementation of the tiny encryption algorithm written in python for a computer and network security class. Our professor had specific rudimentary requirements for how it was to be implemented. I expanded upon the assignment by including testing and extra functionality then refactored. The code is public on my github.
This post is somewhat technical but I will explain everything simply for anyone unfamiliar with the concepts. I will introduce and then explain so bear with me if this looks like gibberish at first.
The tiny encryption algorithm is one of the simplest serious block encryption algorithms. The algorithm works on 64 bit blocks of plain text at a time using a 128 bit key. In the diagram above the plain text is divided into two 32 bit blocks L0 and R0 and the 128 bit key is divided into four 32 bit blocks K0-K3. The formula is represented below.
The Encryption uses repeated application of the pair of rounds shown in the diagram above, defined here for rounds i and i+1 starting with i=1
δ Delta is a predetermined constant 田 denotes addition-mod-232 ⊕ denotes XOR (exclusive or) >> denotes logical shift in given direction
All of the variables are declared as unsigned 32-bit integers, I do this in python by importing c_uint32 from ctypes library. An unsigned 32-bit integer is a binary number with 32 places for 1’s or 0’s and is always positive. The program takes input in hexadecimal base 16 format which extends 0-9 with letters A-F. Meaning F in hexadecimal equals 15 in our standard base 10 and 1111 in base 2 binary.
Hexadecimal is much easier for us humans to deal with because rather than having to write each place (bit) as a 1 or 0 we can represent 4 binary places (4 bits) at a time with 1 hexadecimal digit. This allows us to represent ‘10011111111101010111100111100101’ as 9FF579E5. Convention is to write hexadecimal with a prefixed ‘0x’ so 0x9FF579E5.
A logical shift, >> in the formula as well as within the code, is a bit-wise operation that shifts all of the bits of its operand. This means that all of the places (bits) of a number in binary are shifted.
example of logical shift left by 1 from Wikipedia
Addition-mod-232 means that the values are added together and then divided by 232 and the remainder is the result. Another way of thinking about this is that the two values are added together and if they are more than 232 then 232 is subtracted from the total for the result. The processor performs this differently with overflow and wrapping but this is a simple way to represent the math.
XOR, exclusive or, is represented in the formula as ⊕ and within the code by ^ the caret. XOR is a binary operation that compares bit to corresponding bit and the result evaluates to 1 only if exactly one of the bits is a 1. This means the result is 1 only if one of the bits ‘or’ the other is 1 but not if both are 1. I am reiterating this to differentiate it from the usual logical ‘or’ where it is also 1 if both values are 1.
My implementation is verbose in order to try and grok the algorithm. The assignment was to write a program to perform one pair of encryption rounds and to then decrypt the 64 bit ciphertext which after one pair of rounds is L2 and R2 in the diagram. Also recall that K0-K3 are 32-bit blocks of the 128-bit key.
If not familiar with grok, it is a fantastic term. From Robert A. Heinlein's book Stranger in a Strange Land,
'Grok' means to understand so thoroughly that the observer becomes a part of the observed —to merge, blend, intermarry, lose identity in group experience.
Looking again at the diagram, the first step after getting input and setting up all of the variables is to determine L1 and R1. Following the directional arrows we start by taking R0 into three branches and performing a logical shift left by 4 on one and a logical shift right by 5 on another. The logical shift left branch is then added using addition-mod-232 with K0 and the right branch with K1. The third R0 is also added-mod-232 with Delta1. All three branches are then XOR’ed and the result is added-mod-232 to L0 for the value of R1. L1 is an unchanged R0.
Delta is a predetermined constant. In practice it would usually be a magic number but was determined for the assignment
The next round is essentially the same using the new starting points of L1 and R1 with the next blocks for addition. K2 is added to the logical shift left branch, K3 for the logical shift right branch, and Delta2 for the third branch.
Since the assignment was to only do one round, the ciphertext in L2 and R2 can then be decrypted by reversing the process. Reversing the arrows on the diagram we take L2 to perform the three operations on to XOR together. Then perform a subraction-mod-232 with R2. This gives us R0 and L1 which we then use to determine L0 through the same process but with the earlier key blocks and delta.
In practice it’s recommended to run the algorithm for 64 Feistel rounds (32 pairs of rounds). Feistel refers to the symmetric structure of the algorithm, as clearly seen within the diagram. Also in practice the predetermined Delta would be a magic number, a number chosen to demonstrate that there are no possible hidden properties to it. There are a few weaknesses to TEA which have removed it from common use in cryptography. The code is decent, it’s at 96% coverage on the testing.
After fixing a couple of my old broken android phones, playing around with unlocking the boot-loader, rooting them and installing custom ROM’s I decided to optimize an android for use with Kali NetHunter.
Monitor mode on NetHunter. Detecting two access points with WEP encryption in my neighborhood. These could be cracked in like 8-30 minutes.
Kali Linux NetHunter is an open source android penetration testing platform. Check out Offensive Security’s NetHunter page for more information.
NetHunter has versions available for many android devices. Nexus, OnePlus, Galaxy, Gemini, LG, HTC, some Sony’s and the list goes on. If trying this out make sure to do ample research on the device you are considering because CPU chipset varies on the same android phones between carriers. The wiki on gitlab lists the supported devices.
Also pay attention to the WiFi chipset because only a few inboard WiFi chips can be used for monitor mode, otherwise the phone will need an external adapter for WiFi penetration. To use the inboard chips we’ll have to modify the kernel and firmware but even if using an external adapter be sure to verify compatibility.
I went with the Hawaii Nexus 6P as it supports inboard monitoring mode and Offensive Security sold me with their description for it. Also being able to get one on ebay for 50 something dollars is pretty compelling when some of the other options are still hundreds. (*note the following is all specific to the Hawaii Nexus 6P)
I reinstalled NetHunter while writing this post. These instructions are pretty good, it’s the installation.txt file from:
You need android sdk platform tools fastboot from android studio. Downloading all the files above into the sdk platform tools folder will simplify a lot so that you don’t need to include paths to run fastboot or paths for the files to flash.
If you have issues with the shell script just look within the scripts and ran each command from the shell.
You need to download the twrp recovery image separately, just google the img name.
You will need a usb connected external storage for step 5 because the 6P has no SD card slot.
An add-on for the lineageOS in order to enable root-access to apps is also needed which can be downloaded from the link above. Flash the add-on and then enable developer options and you can grant root access to apps.
XDA developers forum is the best resource I’ve found for all of this. You can find support and instructions for all the other possible devices on their forum as well.
This is a docker swarm on a cluster of raspberry pi’s running OpenFaaS. FaaS stands for Functions as a Service, from Wikipedia it is “a category of cloud computing services that provides a platform allowing customers to develop, run, and manage application functionalities without the complexity of building and maintaining the infrastructure typically associated with developing and launching an app.” Allowing developers to execute code in response to events without having to build out the typical infrastructure.
From Alex Ellis the creator of OpenFaas, “The idea is to make it as simple possible to create a function that is built for, deployed to and run on Docker Swarm or Kubernetes, while providing a workflow that integrates directly with the Docker ecosystem.”
6 raspberry pi 4’s with pimoroni blinkt, gigabit switch, Anker power supply
This allows us to turn anything into a serverless function that runs through Docker Swarm on Linux or Windows. It allows us to upload functions, modular chunks of code, into the cloud and execute independently.
The Pimoroni Blinkt LED’s on the raspberry’s allow for a visual display of the load on each, above is a demo of this scaling up.
Below is a demo of the LED’s showing a rolling update.
And then scaling the service back down to zero.
Headless Raspberry Pi setup
A headless setup allows us to use the raspberry over the network without a monitor, keyboard or mouse. We can install a more lightweight operating system without a GUI (graphical user interface). Saving on hardware costs and the machines resources. The headless Linux install I’m using is based on Debian and designed specifically for Raspberry Pi hardware. You can download it here:
The headless variant as of writing this is Buster Lite. After downloading we need to flash the image onto a microSD card for the raspberry. The easiest way to do this is with balenaEtcher which can be downloaded here:
Etcher automatically unmounts the card so after flashing the image remove then replace the microSD card. To enable SSH you need to create a file entitled SSH on the root partition of the SD named ‘boot’. On mac or Linux you can do this in the command line by navigating to the boot partition and then using command ‘touch ssh’ which will create a file of that name if it does not already exist:
touch ssh
If you are connecting to the pi via Ethernet cable you can skip this next step.
To connect via wifi you need to create another file in the boot partition named:
Change the country code, ssid and psk (pre-shared key) to your own. The password is only stored here in plain text upon first boot, the Pi will move this file automatically.
Adding an ‘ssh’ file to a fresh flash on Raspbian enables Secure Socket Shell on its first boot. This ‘headless’ control over the network doesn’t need a mouse, keyboard or monitor and lets us use a light weight operating system.
The defaults on Raspbian are username: pi and password: raspberry, it’s a good idea to change these.
To SSH into our pi we need to know the IP address. Logging into your router and checking the devices list is an easy way to do so. It can often be reached by browsing to page:
http://192.168.0.1
If you haven’t changed your routers login credentials I strongly suggest that you do so. The defaults can be found located on your router or online.
The IP address of the pi can also be determined using nmap to scan the subnet set to -T4 for greatest speed:
nmap -sV -T4 192.168.0.0/24
Once we know our pi’s IP address we can connect to it via ssh. On windows we may need to install a client to do this, PuTTY.exe is the most commonly used. On Mac or Linux devices SSH clients are installed by default and should run out the box from command line with:
Of course substituting whatever our pi’s IP address actually is. We will then be prompted for a password which is ‘raspberry’ by default. We can then use the command ‘passwd’ to change the password for our default user pi.
I prefer to set my own password for the root account and then relock root as well as disallowing remote root login unless I need it for something. To do so:
sudo passwd root
By default the pi account has passwordless sudo, we can change this by editing the file /etc/sudoers.d/010_pi-nopasswd to be ‘pi ALL=(ALL) PASSWD: ALL’
sudo nano /etc/sudoers.d/010_pi-nopasswd
pi ALL=(ALL) PASSWD: ALL
cntl + x, y, enter
Since we’re running the pi headlessly we can use the command ‘sudo raspi-config’ to change the host name, change the memory split to the gpu to 16, verify/change our locality, enable predictable network names and a bunch more.
It’s a good idea to create a new user and lock user pi although changing to ssh key based authentication is secure regardless. Just to note deleting user pi rather than locking it can cause some issues. If creating a new user it will need to be added to groups:
Then logout of pi into the new user, lock pi, lock root.
sudo passwd -l pi
sudo passwd -l root
For ssh key authentication we need to generate keys if we haven’t before. We can do so on our machine(not ssh’d into the pi) with the following command (for a 4096 bit key):
Disabling password based access to the raspberry prevents anyone without the key from being able to connect. To do so edit /etc/ssh/sshd_config and make sure the password authentication line reads PasswordAuthentication no.
sudo nano /etc/ssh/sshd_config
We can then reboot the pi to make sure all our changes take effect with ‘sudo reboot’
