In the course of a normal day of programming, I often get stuck, learn something new or make progress towards my goals.

Keeping a log seems like a good idea for engineers, it’s good to review what we’ve struggled with in the past and sharing solutions, insights and frustrations with others might make their lives a little easier.

To prepare for my next engineering role, I’ve chosen a four-pronged approach:

  • implement a personal project of mine with a relevant stack. From planning, through implementation and deployment.
  • study algorithms & data structures; the traditional curriculum
  • contribute to an open source project
  • learn about system design

Those goals overlap slightly, e.g. when studying large open source projects, I’ll learn about system design; when trying to improve the performance of my web application, I might have to dig into a more appropriate data structure.

I’m building a web application with Flask, Postgres, Redis, React. As standard as it gets.

I don’t yet know what I’m building, my first goal is to set up a simple application that has all the parts of a production app: it can respond to requests, store data and produce an interface. A generic base if you like. The idea is to have a holistic overview - how does the frontend communicate with the backend? How does the database get deployed? etc. Unconstrained by existing technical descisions, I also get to try some tools I’ve been meaning to learn more about.

unit and integration tests have to be set up, e.g. tests for individual functions but also a fake web client that triggers all routes documentation / a readme is required a solution for secrets (consul?) and configuration is needed.

There are a few repos with applications of this kind, I’m studying them while reading tutorials to get to my generic base.

Once I have this running, I want to deploy it and learn about setting up a build-pipeline, afterwards I can add more features.


To isolate the components from my OS, I’m running Postgres and Redis within Docker containers. The idea here is to avoid problems of interference of several projects on my machine, they might need different versions of Redis, the binary I get through pacman (on Arch) might not be the same as what I’m getting when running Alpine on Ubuntu or CentOS within Digital Ocean or AWS.

The containers are in turn controlled by docker-compose.

I couldn’t launch the docker daemon on my arch machine.

I deleted docker and reinstalled it, that didn’t help. I deleted all docker related files in /var/lib/docker and /etc/docker

I added docker to my groups and myself into the group to not have to run docker with admin privilges

sudo groupadd docker sudo usermod -aG docker

I tried starting docker via systemctl

sudo systemctl start docker.service

I tried to read the output of journalctl-xe, but that wasn’t helpful

in the end, there was the tipp that on Arch I just had to reboot.

That worked. Now Docker runs on my machine!

A lot of software engineering is fighting with operating system configs. Docker, Databases, the network. The projects are so complex, a million things can go wrong. Not exactly engineering to google around aimlessly for hours. That’s the lot of the programmer who just glues together complex software projects.

I finally got Redis within Docker to work, the trick is to use port forwarding, otherwise the service is not available

playing a bit with the redis-cli is useful, to understand what Redis can do

I should have cheat sheets ready for: - Python - Go - JS - Redis - Postgres - Docker - Unix/Linux. systemd, common utilities - git: how to get out of fucked up situations: cherry picking, reverting etc - Vim … no reason to have all of this information in mind

The static keyword in C has two uses:

Within a function, a value can be maintained between invocations

f() { a = 5 static b = 5 a += 10 b += 10

return a, b

}

f() // 15, 15 f() // 15, 25 f() // 15, 35

Should be used with care, it’s harder to reason about.

For variables and functions in a file (a compilation unit), the static keyword makes the function or variable only accessible within the file, not to the outside world. It’s for encapsulation and a good practice.

the strchr() function in C: –> char* strchr(char* a_string, char a_char) it returns the location (the pointer) of the first occurence of a_char in a_string or NULL, if a_char isn’t in a_string

The role of the modern engineer is to combine complex software components intelligently. Software modules, like Python libraries, infrastructure like AWS and free-standing components like databases can be combines with services (available through APIs). It’s important to note, that humans should interact with these systems, too. Instagram, Facebook, … these are nice traps to catch people in.

Can the complexity be further reduced? Hosted databases are a great step, more restrictive programming languages help.

I haven’t built anything interesting, even though I have all the resouces and building blocks available to me. Experts can be contacted, books can be read. What am I waiting for?

it feels like a lot of Python programmers have gone to Rust; lots of C programmers have gone to Go.

Web apps are not supposed to hold any state. If that’s the case, then many app servers can be spun up and serve requests given to them by a load-balancer. 10 instances of a web app could then speak to one database.

I like pipenv. It ties together Pip and Virutalenv –> the idea is to not rely on Python packages that exist on the system, but to create an isolated virtual environment that contains all the packages. This way, there aren’t any conflicts between packages.

So far, the app is using: Docker (and docker-compose), Postgres + the connector, Redis + the connector, Python, flask, sqlalchemy… An incredibly amount of code for some tiny functionality. And it’s going to get much more extreme soon. With API calls to external services, deployment and the use of python packages. I need to set up linters and tests! Get inspiration from other flask projects.

Nice tutorials, good documentation and sample code make this situation bearable.

Electronics, embedded systems and computer engineering is an entire world in itself. Hundreds of thousands of professionals making our modern world possible. I manage my python versions with pyenv

when I run “info” within the redis-cli, the value I get for the git hash is just “0000000”

Through some grepping within the redis codebase, I found the file “/src/mkrelease.hdr”, on line 2

in the commandline, using the logical or operator “||” like this || means, “Take the result of if the exist status shows no error, and otherwise.

Honestly, glueing together several complex software components, configuring them and deploying them is pretty disheartening. I’m not programming, I don’t really learn anything of general relevance, I just fight my way from one error message to the next, googling widely and eventually finding some stupid solution on Stack Overflow or in a Github issue thread.

When reading in the Redis source, I also stumbled upon /dev/null, which can be used to discard output:

https://github.com/antirez/redis/blob/unstable/src/mkreleasehdr.sh

GIT_SHA1=`(git show-ref –head –hash=8 2> /dev/null || echo 00000000) | head -n1

https://stackoverflow.com/questions/10508843/what-is-dev-null-21