I recently wrote a better viewer for an earlier post I have on this blog. Now you can simulate discovering new technology without ever loading up Civiliation VI!

As a big fan of Civilization VI, I recently decided to create a program that displays a random quote from the game, inspired by the iconic messages that accompany each new technology discovery. This viewer builds on an earlier post I wrote here, but now it’s more interactive and immersive, making it feel like you're advancing through the tech tree without even having to open the game.

Learning kanji can feel overwhelming, but here’s something interesting: with just 777 carefully selected kanji, you can understand over 90% of the characters you'll encounter in everyday Japanese. These kanji, originally adopted from mainland Asia over centuries, have woven themselves into the very fabric of the Japanese language, forming the backbone of countless verbs and nouns.

If you aim for native-level proficiency, mastering kanji isn’t optional—it’s essential. But rather than slogging through 2,400 random characters, why not focus on the most common 800 and get the bulk of the language covered? This is the thinking behind the kanji trainer I built. It's designed to help you prioritize your learning intelligently, much like Japanese Complete, which teaches stroke order, meaning, and mnemonic breakdowns. My tool serves a similar purpose, offering a minimalist, straightforward approach to mastering kanji efficiently.

The Design

The page's design is deliberately minimal. Each kanji is presented in large font, demanding attention. The learner must first engage with the character alone before clicking to reveal its meanings, hiragana, and romanji. This design echoes a principle found in modernist art: let the essential object stand, unadorned, in its own right. In this way, the kanji remains the focal point until the learner decides to dig deeper.

While looking for a solution to display a map embedded into a web page, I came across several options. My initial idea was to use something like Google Maps or Bing Maps. Both wanted me to sign up for a developer account to get access to their APIs. Additionally, they often impose extreme rate limiting or superimpose intrusive watermarks unless you upgrade to a paid account.

OpenStreetMap seemed like a great alternative. Over the past few years, its maps have greatly increased in quality. Its open licensing and public tileset make it easy to include in any project. I initially struggled to find any quick examples of how to utilize it in browser with Javascript, but I came across Leaflet.js and it seemed like a great fit. It is open source, lightweight, and mobile ready.

Below is a quick example of what I ended up using it for.  You initialize the map with a location ([38, -97] is the geographic center of the US), zoom level (4 seems to fit the entire US), and a tileset (OpenStreetMap). All that is needed to add a marker is a to call the marker function with decimal GPS coordinates.

I've been using Python after a dry spell of using other languages. I went through the basic mechanics of the language and tried to use a wide variety of different techniques to cover most of the usage. Here's what I came up with.

from random import random, randint, choice, shuffle
from math import log, log10, log2
from itertools import count, repeat, cycle, chain, islice
import re


print('lists')
l = [1, 2, 4, 3, 5]
print(l)
l.append(6)
l.extend([8, 7])
print(l)
print(sorted(l, reverse=True))
print(reversed(l))
for i, x in enumerate(l):
    print(i, x)
print()

print('dicts')
d = {'a': 1, 'b': 2, 'c': 3}
print(d, d.keys(), d.values())
print(d.get('a'), d['b'], d.get('z'))
d.setdefault(0)
print(d.get('z'))
d.update({'d': 4})
print(d)
print(dict(zip(range(9), l)))
print()

print('sets')
s = set([1, 1, 1, 2])
s.add(3)
s.remove(3)
s.discard(55)
print(s, s.union([4]))
print(s.difference([1]))
print(s.intersection([1]))
print()

print('regex')
m = re.search(r'(\d{4})-(\d{2})-(\d{2})', '2016-01-01')
print(m.groups())
res = "1a2b3c"
newres = re.sub(r'[a-z]', ' ', res)
print(newres)
print()

print('itertools')
print(list(islice(count(start=0, step=1), 2)))
print(list(repeat(5, 3)))
print(list(chain([1, 2, 3], [4, 5, 6])))
print(list(islice(cycle([1, 2, 3]), 7)))


def count2(start, step):
    while True:
        yield start
        start += step


print(next(count2(6, 2)))
print()

print('type')
print(type(5), isinstance(5, int))
print()

print('string')
s = ' asdf,abc#'
print(s.strip(), s.strip('#'))
print(s.split(','), ','.join(str(x) for x in [1, 2, 3]))
print(s.find(','), s.find('x'))
print('{}: ${:3,}'.format('Cost', 1234))
print()

print('stdin')
# if __name__ == '__main__':
# jptr = open(os.environ['OUTPUT_PATH'], 'w')
# s = input()
# result = isValid(s)
# fptr.write(result + '\n')
# fptr.close()
print()

print('numbers')
s = '123'
i = int(s)
print(pow(i, 2), abs(-i), round(i + .94423, 3))
print(log2(i))
r_float = random()
r_int = randint(0, 10)
arr = [1, 2, 3]
r_el = choice(arr)
shuffle(arr)
print(r_float, r_int, arr, r_el)
print(~100, 101 & 9, 101 | 9)
print()

print('date')
from datetime import date, time, datetime, timedelta
now = datetime.utcnow()
print(now)
print()

print('list comprehension')
l = [i+1 for i in range(5)]                   # [1, 2, ..., 10]
s  = {i for i in range(10) if i >3}            # {6, 7, 8, 9}
iter = (i+5 for i in range(5))                   # (5, 6, ..., 14)
d = {i: i*2 for i in range(5)}                # {0: 0, 1: 2, ..., 9: 18}
from pprint import pprint
print(l, s, next(iter), d, any(x % 2 == 0 for x in iter))
pprint(d, width=10)
print()

print('lambda and enum')
from functools import reduce
iter = map(lambda x: x + 1, range(10))            # (1, 2, ..., 10)
iter2 = filter(lambda x: x > 5, range(10))         # (6, 7, 8, 9)
o  = reduce(lambda out, x: out + x, range(10))  # 45
print(next(iter), list(iter2), o)
from enum import Enum
Direction = Enum('Direction', 'n e s w')
print(Direction, Direction.n)
print()

print('try catch')
class CustomEx(Exception):
    pass

try:
    print('try')
    raise CustomEx('msg')
except CustomEx as e:
    print('except', e)
finally:
    print('finally')
print()

In my last post, I was bulk gathering GPS coordinates from the internet. After I had them, I still needed a method to display them. Luckily, Google My Maps offers a versatile import function. Every new layer can bulk import data from a variety of data formats.

On Wikipedia, coordinates are displayed in the degree-minute-second format. Since Google Maps seems to favorite the degree-decimal format, I was uncertain if importing would convert them to the correct location.

Name,Location for Google
"Alpha, California","39°19′50″N,120°46′53″W"
"Badger Hill, California","39°12′58″N,121°02′58″W"
"Baltimore Town, California","39°23′12″N,120°31′51″W"

The import worked without any issues in with this format.  You can import CSV, XLSX, KML or GPX with Google Maps. Google Maps will prompt you for the meaning of each of these fields after you import the CSV. It will even ask you for the format of the coordinates (latitude,longitude vs longitude,latitude).

In order to convert this location data into KML, all I needed to do was export it. The export function can be found in the options for the entire map. You can choose which layers to include in the dialog.

KML is an XML based format. Once you export the map, you can open it up in any text editor to see the data it produced.

<Placemark>
    <name>Alpha, California</name>
    <description>39°19′50″N,120°46′53″W</description>
    <styleUrl>#icon-seq2-0-0-0288D1-labelson</styleUrl>
    <Point>
    <coordinates>
    -120.781388888889,39.3305555555556,0
    </coordinates>
    </Point>
</Placemark>

If you want to try it out for yourself, you can use the same map I referenced in this article: Nevada County Ghost Towns

I've written a lot of webscrapers in the past. Each approach I've used has its benefits and drawbacks. If I need to be able to do anything more complicated than extract some text from an HTML document, Nightmare has been my favorite way to do it. It's easy to use, fast, lends itself to readable code, and comes with a fully featured headless browser with JavaScript.

Most recently, I was researching a hobby of mine, finding ghosts towns to visit. Wikipedia is a great resource for them. It's a more comprehensive list than I could find anywhere else and many of the towns had GPS coordinates. However, what I really wanted was a map so I could get an idea where each town was in relation to the others. Since I couldn't find it anywhere, I thought I would map it out myself with a custom Google Map. The prospect of copying and pasting each of the 100+ towns into Google Maps seemed daunting.

So I decided to automate it. First I would need to get the list ghost towns and their respective URLs on Wikipedia. Then I could grab the coordinates from that page and put it into a CSV file that Google Maps could import. Simple enough.

Installing Nightmare is as easy as running:

npm install nightmare

It will install a version of Electron in your node-modules folder. Whenever you initialize your script with Nightmare, it will run your scraping code inside of sandboxed browser.

const nightmare = require('nightmare');

// Run Nightmare in an async function to take advantage of 
// ES7's await
async function run() {
  // Set large timeout of 150 seconds, default is 30
  const nm = nightmare({
    waitTimeout: 150000,
  });
    
  // Grab the list of towns from the county webpage
  const towns = await nm.goto(`https://en.wikipedia.org/wiki/Template:Nevada_County,_California`)
    .wait('body')
    .evaluate(function() {
      const ghostTown = document
        .querySelector('tr > th > a[title="Ghost town"]');
      const row = ghostTown.parentElement.parentElement;
      // get the URL for each town from its link
      const towns = Array.from(row.querySelectorAll('li > a'))
        .map((el) => el.href);
      return towns;
    })
    .catch(function(error) {
      console.error('did not get towns', error);
    });
    
  // Set a delay of 1 second per page, to play nice with
  // Wikipedia's rate limiting
  let delay = 1000;
  const res = towns.map(async function(url) {
    // increasing this delay will stagger each promise to 
    // roughly execute 1 second after each other
    delay += 1000;
    const coord = await nm.wait(delay)
      .goto(url)
      .wait('body')
      .evaluate(() => {
        // scrape latitude and longitude data
        const lat = document
          .querySelector('span[class="latitude"]').innerHTML;
        const long = document
          .querySelector('span[class="longitude"]').innerHTML;
        return {lat, long};
      })
      .catch((error) => {
        console.error(`did not get coords: ${url}, ${error}`);
      });
    return {
      url: url,
      coord: coord,
    };
  });
  const coords = await Promise.all(res);
  // Only display towns that have coordinates
  console.log(coords.filter((c) => c.coord !== undefined));
  await nm.end();
}

run();

Using Nightmare is a lot like scripting a browser. You initialize it with nightmare() which is a lot like starting up the browser. Visiting new pages can be achieved by calling its .goto(url) function. You can chain any number of these actions and the Election instance will evaluate them sequentially. .evaluate(function) allows you to execute Javascript code within that instance. It gives you everything you'd expect in a browser console, including the document and window object.

I ran into some issues with Wikipedia's rate limiting as I was fetch over 100 pages asynchronously. To get around this, I  used Nightmare's .delay(ms) functionality. This will pause the browser for that specified amount of time. I did it before even going to the page to prevent too many sequential page loads.

Nightmare is uses asynchronous functions at its core. Almost every operation done with it returns a promise. It's a great use case for ES7 await keyword. Without it, you start to get into some fairly nasty nested function which will make the code a lot less readable.

As a last note, if you are running this on a Linux machine without X installed like I was, Electron will silently fail. To get around this I installed xvfb and ran my script with xvfb-run node ghost-town-scraper.js.

For years, I've been a big fan of DuckDuckGo's bang feature. I use it any time I have a specific site I want to search. For example, if I know the result I want to get is on Wikipedia, you can simply use !w [search term] rather than having to see a search result page or having to go to Wikipedia and using their search box. You wouldn't think the extra click matters, but it adds up.

However, with DuckDuckGo, if you do not put a ! at the beginning of your search, it will act like any other search engine and show you a page of results.

Google is still the king of getting relevant search results, so I often find myself switching between the two. So much so, I used DuckDuckGoog which provides google search results if you do not use the bang feature but redirects you to DuckDuckGo if you do. I've found it to be the best of both worlds.

Today, DuckDuckGoog is down. It's a fairly straightforward hack to check if your search query starts with a !, so I decided to host my own. I had never done so in the past because it seemed like such a trivial service to host. It hardly seems worth it to load up a PHP script or redirect to a NodeJS page.

I settled on using NGINX's Lua scripting capabilities. It seemed like the right choice because it gets you as close to the webserver as possible but frees you from the limitations of NGINX's config language. It's available in the nginx-extras package on any debian derived system like Ubuntu or Mint.

You'll need to place this section of config of whatever host you are serving it from.

NGINX configuration

location /search {
  include /path/to/duckduckgoog.lua;
}

Here's the script I came up with. It needs to be placed in a file the webserver has access to.

Lua

access_by_lua "
local url = ngx.var.uri
local args = ngx.var.args

function starts(str, start)
   return string.sub(str, 1, string.len(start)) == start
end

function split(str, pat)
   local t = {}  -- NOTE: use {n = 0} in Lua-5.0
   local fpat = '(.-)' .. pat
   local last_end = 1
   local s, e, cap = str:find(fpat, 1)
   while s do
      if s ~= 1 or cap ~= '' then
         table.insert(t,cap)
      end
      last_end = e+1
      s, e, cap = str:find(fpat, last_end)
   end
   if last_end <= #str then
      cap = str:sub(last_end)
      table.insert(t, cap)
   end
   return t
end

local q = split(args, '=')[2]

if starts(q, '!') then
    local ddg_url = 'https://duckduckgo.com/?q='
    -- print(ddg_url .. q)
    ngx.redirect(ddg_url .. q, 301)
else
    local goog_url = 'https://www.google.com/search?q='
    -- print(goog_url .. q)
    ngx.redirect(goog_url .. q, 301)
end
";

I ran into a problem when I was trying to display an HTML page hosted on Github. Since Github serves HTML files as raw text in the content type headers, you can't just browse to the raw file in a repo and view it. Usually RawGit works great as a shim, but I have found unpredictable behavior on larger html files, especially when they are Javascript heavy.

I wanted to embed this content on another webpage and it seemed like frames were a good match for what I wanted to accomplish. Frames usually take a URL in the src attribute to load there content. This was a problem because if I gave it the URL to the raw file, it would still honor the content type header and display it as raw text instead of HTML.

I came up with a way of creating an empty I frame and simply setting the content after it is created. You can do this using the DOM method document.writeln():

const setFrame = (el, html) => {
  if (el.contentDocument) {
    el.document = el.contentDocument;
  }
  if (el.document !== null) {
    el.document.open();
    el.document.writeln(html);
    el.document.close();
  }
};

Here's a working example:

The C++ standard libary has come a long way in the last ten years. It contains a lot of useful functions that make stuff easy that used to be a chore in C++ compared to other languages. However, there is still a gap between what the STL provides compared to other modern languages.

One such case is UUID/GUID creation. In languages like Python and Java, to make a GUID, all you need to do is import a module and you are good to go. In C++, this will still be relegated to an external framework like the Windows API or QT. There are even some smaller cross platform libraries specifically targeting this functionality. In most cases, this is still the best place to go. These methods are battle-tested: they'll be fast and robust.

If you want something more portable that doesn't rely on any external linking, you'll have to write it yourself. Luckily, the UUID format isn't particularly complex. If you can generate random hex characters, you can make a GUID easily.

The STL in C++11 now includes std::random which makes this a lot easier. Before it, you would still probably need to go to an external library to get random values. In combination with std::hex, almost all of the hard parts of this problem can be solved with stuff in the standard library.

#include <vector>
#include <iostream>
#include <sstream>
#include <random>
#include <climits>
#include <algorithm>
#include <functional>
#include <string>

unsigned char random_char() {
    std::random_device rd;
    std::mt19937 gen(rd()); 
    std::uniform_int_distribution<> dis(0, 255);
    return static_cast<unsigned char>(dis(gen));
}

std::string generate_hex(const unsigned int len) {
    std::stringstream ss;
    for(auto i = 0; i < len; i++) {
        auto rc = random_char();
        std::stringstream hexstream;
        hexstream << std::hex << int(rc);
        auto hex = hexstream.str(); 
        ss << (hex.length() < 2 ? '0' + hex : hex);
    }        
    return ss.str();
}

These two functions allow you to generate hex random characters of n length. From there it is as simple calling it for each of the 5 parts of a UUID by their respective lengths. If you can easily access the UUID functionality in another library, use it. It is likely more performant and can take full advantage of the OS APIs.

repl.it link

Most programming languages today have great JSON support. If it isn't built into the language, there is a de facto standard library you can add to your project. Included in these libraries is a formatting function. For the sake of space, most of these libraries will have your JSON minified by default. For larger documents, this quickly becomes very unreadable to users.

Because of this, these libraries have a JSON format function. However, it isn't a terribly hard problem to solve. If all you need to do is format a JSON document that is already a string, many of these libraries would force you to parse the document into memory and then write it out.

Below is a function that formats any valid JSON string. By default, it indents it by two spaces but that is fully parameterized in this function. It is written in C# because it is one of the few popular languages that relies on a 3rd party library for its JSON functionality. This could also easily be rewritten in almost any imperative language.

public static string FormatJson(string json, char indent = ' ', int times = 2)
{
	var indentation = 0;
	var quoteCount = 0;
	var indentString = string.Concat(Enumerable.Repeat(indent, times));
	var result = "";
	foreach (var ch in json) {
		switch (ch) {
			case '"':
				quoteCount++;
				result += ch;
			break;
			case ',':
				if (quoteCount % 2 == 0) {
					result += ch + Environment.NewLine + 
                    string.Concat(Enumerable.Repeat(indentString, indentation));
				}
			break;
			case '{':
			case '[':
				var openFormatted = string.Concat(Enumerable.Repeat(indentString, ++indentation)) 
                + indent;
				result += string.Format("{0}{1}{2}", ch, Environment.NewLine, openFormatted);
			break;
			case '}':
			case ']':
				var closeFormatted = string.Concat(Enumerable.Repeat(indentString, --indentation));
				result += string.Format("{0}{1}{2}", Environment.NewLine, closeFormatted, ch);
			break;
			default:
				result += ch;
			break;
		}
	}
	return result;
}

I was coding in a C# project when I came across a class that was mapping names to GUIDs. Since GUIDs are objects, you can't simply put them in a const array or an enum like you can do with simpler data types.

public static class Guids
{
	public static readonly Guid First = new Guid("F288114A-B29E-40A2-B649-DF87B10B8E7C");
	public static readonly Guid Second = new Guid("AFE63E51-A611-426C-8226-4A98A93195DB");
	public static readonly Guid Third = new Guid("02CFDDDE-5C0E-4E65-B756-25DB72A08503");
	public static readonly Guid Fourth = new Guid("500CDF93-D1D1-4A89-A348-5A1B5265B1FA");
	public static readonly Guid Fifth = new Guid("890CD9E4-59B7-4C9B-95BF-9FFC159E169E");
}

If I was trying to look up a GUID by name, it would be as easy as accessing the object by the name, but I had a GUID and wanted to get the same for it.

The approach I took was to create a dictionary where the GUID values were the keys and the names were the values. Using the GetFields() method on the type of the object seemed like best way to get the public properties. I could then use LINQ to convert this list of fields to the data structure I needed.

I found myself recently having to deal with a large and complex XML document. I wanted to use Python's excellent paradigms like generators and list comprehension to carve up the XML into more meaningful data. However, most libraries that deal with XML parsing provide you with non-standard interfaces to access XML nodes due to the complexity of the XML spec.

While searching for a solution, I came across the xmltodict library. It turned out to be exactly what I needed. I was surprised at the intuitiveness of the API. Essentially, give it an XML document and it gives you a native Python dictionary with intuitive mappings for XML specific features like attributes and namespaces. For example, lists in the form of <item>1</item><item>2</item><item>3</item> simply become item: [1, 2, 3]. Once it is in Python's dictionary model, it is trivial to get it to convert to JSON due to their similarity.

Below is a working snippet of a few test cases I came up with to demonstrate its functionality.

If you need to create HTML, it is usually a bad idea to just concatenate strings together. It tends to create brittle and invalid documents once there is any level of complexity to it. You need to manage the position of each tag and when to close it, where to insert data, and changing the formatting is a chore. Since HTML is a tree-like data structure, it is much better to model it as such inside your program.

The best option is usually to a bonifide HTML parser/generator. Most languages either include it in the standard library or has a defacto external library that most users work with. When I was working with C#, I found that I needed to generate a simple HTML document but found that there was nothing in standard library explicitly for it. However, C# does have excellent standard library support for XML.

While HTML is not XML, their formats are similar enough that if you know the differences, XML generators can create valid HTML documents easily. For my purposes, installing an external HTML generator would have been an irrelevant dependency on my project, so the built in System.Xml.XmlDocument seemed a better fit.

Below is a code snippet that will create a basic, valid HTML document as a string. I didn't want to include anything XML specific like namespaces or CDATA tags, so I specifically omitted solutions that required additional configuration to get rid of these. Note that you need to use doc.outerXml to get the string data from the object. The native doc.toString() will not convert the document to a string.

using System.Xml;
public string GenerateHtml()
{
	var doc = new XmlDocument();
	var htmlRoot = doc.CreateElement("html");
	doc.AppendChild(htmlRoot);
	var body = doc.CreateElement("body");
	body.AppendChild(doc.CreateTextNode("Hello world"));
	htmlRoot.AppendChild(body);
	return doc.OuterXml;
}

This blog is running Ghost, a relatively new blogging platform developed with NodeJS. It seemed like a perfect match for this website when I was first starting off because I only needed something lightweight and easy to manage. I also appreciated it being in JavaScript, the main language I use to program these days. If I ever needed to hack it, it would certainly be a lot more fun to work with than a lot of the long-in-the-tooth PHP CMSes out there.

When I was looking into gathering some information on my posts, I found Ghost has a ready-to-go JSON REST API readily available. To use it, you must first enable it in settings:

Then it is as simple as requesting this URL https://YOUR_DOMAIN/ghost/api/v0.1/posts/?client_id=ghost-frontend&client_secret=xxxxxxxxxx. The domain and the client secret will change from installation to installation. The 'client secret' is publically available on all Ghost pages. You can find it by viewing the source of any Ghost generated page or by looking in the database of your Ghost installation.

Below, I've included a quick example using Axios and React using this URL. I've also included the limit option as Ghost by default only returns the first page of results.

After writing my latest post on scraping text from Amazon's web Kindle reader, I got an idea. I wondered how hard it would be to scrape an entire book with the method.

Here's what I came up with:

function hashString(str){
	let hash = 0;
	for (let i = 0; i < str.length; i++) {
		hash += Math.pow(str.charCodeAt(i) * 31, str.length - i);
		hash = hash & hash; // Convert to 32bit integer
	}
	return hash;
}

var hashes = {};
var content = [];

function addDiv(div){
	let hash  = hashString(div.innerText);
	if (hashes[hash] === undefined) {
		hashes[hash] = true;
		content.push(div.outerHTML);
	}
}

var timeout = null;
function main() {
	var appFrame = document.querySelector('#KindleReaderIFrame').contentDocument;
var contentFrames = Array.from(appFrame.querySelectorAll('iframe')).map(f => f.contentDocument);
	Array.from(contentFrames[1].querySelectorAll('body > div')).forEach(addDiv);
	timeout = setTimeout(main, 1000);
	appFrame.getElementById('kindleReader_pageTurnAreaRight').click()
	console.log('content', content.length);
}
main();

Essentially, this script will turn the page every second and look for any new text in the frame the Kindle app uses to display the book. The frame is more or less a window of text to buffer. It doesn't just contain the text displayed on the screen. There is no guarantee that going to the next page would add new content to the frame. However, the app does ensure that only how paragraphs get added or removed and these paragraphs are in order. I reused the string hash code function from an earlier post as an efficient way to see if the string had already been added.

The end result is an array where each element is a paragraph of text. If you let this run through the entire book, this array will have the entire contents of the book in HTML.