Mapnik Generative AI workflow: Processing¶
Alexander Dunkel, Madalina Gugulica, Institute of Cartography, TU Dresden
Mapnik rendering based on stable diffusion generative AI and social media data.
This notebook is a continuation from the previous notebook (01_mapnik_generativeai.html).
Prepare environment¶
Load base dependencies:
import os, sys
import re
import random
import shutil
import geopandas as gp
import pandas as pd
import geopandas
import matplotlib.pyplot as plt
import rasterio as rio
from pathlib import Path
from rasterio.plot import show
Install temporary package rembg
!../py/modules/base/pkginstall.sh "rembg"
Import every cell from the previous notebook, except those tagged with active-ipynb. This will make all variables and methods from the previous notebook available in the current runtime, so we can continue where we left.
module_path = str(Path.cwd().parents[0] / "py")
if module_path not in sys.path:
sys.path.append(module_path)
from modules.base import raster
from _02_generativeai import *
Symlink font folder
!rm /fonts && ln -s {TMP}/fonts /fonts
Symlink remove white background model:
!ln -s {OUTPUT}/isnet-general-use.onnx /root/.u2net/isnet-general-use.onnx 2> /dev/null
Create new directories
dlist = [
(OUTPUT / "images_gis"),
(OUTPUT / "img2img"),
(INPUT / "cluster_img"),
(INPUT / "cluster_img_tags"),
]
for folder in dlist:
folder.mkdir(exist_ok=True, parents=True)
Activate autoreload of changed python files:
%load_ext autoreload
%autoreload 2
Parameters¶
APIURL = "http://127.0.0.1:7861"
BASE_PROMPT_POS: str = \
"white background,simple outline,masterpiece,best quality,high quality," \
"<lora:Japanese_style_Minimalist_Line_Illustrations:0.2>"
BASE_PROMPT_NEG: str = \
"(bad-artist:1),(worst quality, low quality:1.4),lowres,bad anatomy,bad hands," \
"((text)),(watermark),error,missing fingers,extra digit,fewer digits,cropped,worst quality," \
"low quality,normal quality,((username)),blurry,(extra limbs),bad-artist-anime," \
"(three hands:1.6),(three legs:1.2),(more than two hands:1.4),(more than two legs,:1.2)," \
"label,(isometric), (square)"
Set global SD-settings
payload = {
"CLIP_stop_at_last_layers": 1,
"sd_vae":"vae-ft-mse-840000-ema-pruned.safetensors",
"sd_model_checkpoint":"hellofunnycity_V14.safetensors",
}
requests.post(url=f'{APIURL}/sdapi/v1/options', json=payload)
Have a look at our per-job basis settings, loaded from the last notebook:
SD_CONFIG
For this notebook, increase steps to 28
SD_CONFIG["steps"] = 28
Test image generation for tags and emoji¶
The next step is to process social media metadata (tags, emoji) in descending importance (cluster-size), generate images for clusters, and place images on the map, according to the center of gravity for the cluster shape from tagmaps package.
Test API for selected tags¶
PROMPT = "(Grosser Garten, Palais, Nature)"
output_name = "test_image_palais_default"
KWARGS = {
"prompt": concat_prompt(PROMPT),
"negative_prompt": BASE_PROMPT_NEG,
"save_name": output_name,
"sd_config": SD_CONFIG,
"show": False
}
DKWARGS = {
"resize":(350, 350),
"figsize":(22, 60),
}
if not (OUTPUT / "images" / f'{output_name}.png').exists():
generate(**KWARGS)
imgs = list((OUTPUT / "images").glob(f'{output_name}*'))
tools.image_grid(imgs, **DKWARGS)
We have to think about a way to better incorporate these square images in the map. Maybe if we add A thought bubble of to our prompt?
def generate_samples(
prompt: str, save_name: str, kwargs=KWARGS, output=OUTPUT,
dkwargs=DKWARGS, print_prompt: bool = None, rembg: bool = None):
"""Generate and show 4 sample images for prompt"""
kwargs["prompt"] = concat_prompt(prompt)
if print_prompt:
print(kwargs["prompt"][:50])
kwargs["save_name"] = save_name
if not (output / "images" / f'{kwargs["save_name"]}.png').exists():
if rembg:
generate_rembg(**kwargs)
else:
generate(**kwargs)
imgs = list((output / "images").glob(f'{kwargs["save_name"]}*'))
tools.image_grid(imgs, **dkwargs)
generate_samples("(thought bubble of Grosser Garten, Palais, Nature)", save_name="test_image_palais_bubble")
or maybe icon?
generate_samples("(A map icon of Grosser Garten, Palais, Nature)", save_name="test_image_palais_icon")
Let's keep A map icon of as the pre-prompt.
Some more tests for other tags and terms
generate_samples("(A map icon of Botanischergarten), flower, grün, 🌵 🌿 🌱", save_name="test_image_botan_icon")
generate_samples("(A map icon of Gläsernemanufaktur), volkswagen, building", save_name="test_image_vw_icon")
generate_samples("(A map icon of zoo), zoodresden, animals", save_name="test_image_zoo_icon")
generate_samples("(A map icon of fussball stadion), dynamo, stadion", save_name="test_image_fussball_icon")
generate_samples("(people 🏃), activity", save_name="test_image_running_activity")
Enough tests. Now, we can move to collecting tag and emoji clusters and move on to batch generation.
Process clustered data¶
The overall workflow looks like this:
- Find all clusters above a weight of
x - Walk through clusters, get cluster centroid
- Select all other cluster-shapes that can be found at this location
- Concat prompt based on ascending importance
- Generate image, remove background, save
- Create Mapnik Stylesheet to place images as either symbols or raster images
- Render map
- (Adjust parameters and repeat, until map quality is acceptable)
data_src = Path(INPUT / "shapefiles_gg" / "allTagCluster.shp")
gdf = gp.read_file(INPUT / "shapefiles_gg" / "allTagCluster.shp", encoding='utf-8')
CRS_PROJ = gdf.crs
def sel_cluster(gdf: gp.GeoDataFrame, min_weight: int) -> gp.GeoSeries:
"""Return GeoSeries of Clusters above min_weight"""
with fiona.open(data_src, encoding='UTF-8', mode="r") as shapefile:
for feature in shapefile:
properties = feature["properties"]
if properties["HImpTag"] == 1 and properties["ImpTag"] == feature_name.lower():
bounds = shape(feature["geometry"]).bounds
if add_buffer:
bounds = add_buffer_bbox(bounds, buffer = add_buffer)
return bounds
OUTPUT_MAPS = TMP / "bg"
Reproject raster
%%time
raster.reproject_raster(
raster_in=f"{OUTPUT_MAPS}/grossergarten_carto_17.tif",
raster_out=f"{OUTPUT_MAPS}/grossergarten_carto_17_proj.tif",
dst_crs=f'epsg:{CRS_PROJ.to_epsg()}')
basemap = rio.open(f"{OUTPUT_MAPS}/grossergarten_carto_17_proj.tif")
bbox_map = gdf.total_bounds.squeeze()
minx, miny = bbox_map[0], bbox_map[1]
maxx, maxy = bbox_map[2], bbox_map[3]
x_lim=(minx, maxx)
y_lim=(miny, maxy)
Plot all cluster shapes
fig, ax = plt.subplots(figsize=(10, 10))
rio.plot.show(basemap, ax=ax)
gdf.plot(ax=ax, facecolor='none', edgecolor='red', linewidth=0.1)
ax.set_xlim(*x_lim)
ax.set_ylim(*y_lim)
ax.set_axis_off()
Plot only cluster shapes above a certain weight
cluster_sel = gdf[gdf["Weights"]>300]
def plot_clustermap(cluster_sel: gp.GeoDataFrame, basemap: rio.DatasetReader, label: bool = None):
"""Plot a map with clusters, basemap, and cluster labels"""
if label is None:
label = True
fig, ax = plt.subplots(figsize=(7, 10))
rio.plot.show(basemap, ax=ax)
cmap=plt.get_cmap('Paired')
cluster_sel.plot(ax=ax, facecolor='none', cmap=cmap, linewidth=1)
if label:
tools.annotate_locations_fit(
gdf=cluster_sel, ax=ax,
text_col="ImpTag", arrowstyle='-', arrow_col='black', fontsize=10,
font_path="/fonts/seguisym.ttf")
ax.set_xlim(*x_lim)
ax.set_ylim(*y_lim)
ax.set_axis_off()
with warnings.catch_warnings():
# Ignore emoji "Variation-Selector" not found in font
warnings.filterwarnings("ignore", category=UserWarning)
plt.show()
plot_clustermap(cluster_sel=cluster_sel, basemap=basemap)
There are several clusters visible. On the upper left, we can see the Dynamo Dresden stadium. Several tag and emoji cluster shapes are can be found at in this area. There is also a big shape covering the Großer Garten. Two smaller shapes can be found hovering the Dresden Zoo and the Gläserne Manufaktur.
Process Emoji¶
We start with processing emoji. This seems like the easier part, since emoji are already highly abstracted concepts that can convey many meanings in a simplified form.
Some emoji, however, are very generic and used for arbitrary context. We use a broad positive filter list with 693 emoji (out of about 2000 available) to focus on specific activity and environment emoji.
emoji_filter_list = pd.read_csv(
INPUT / 'SelectionList_EmojiLandscapePlanning.txt', header=None, names=["emoji"], encoding="utf-8", on_bad_lines='skip')
emoji_filter_list = emoji_filter_list.set_index("emoji").index
print(emoji_filter_list[:20])
cluster_sel = gdf[(gdf["Weights"]>100) & (gdf["emoji"]==1) & (gdf["ImpTag"].isin(emoji_filter_list))].copy()
plot_clustermap(cluster_sel=cluster_sel, basemap=basemap)
We can see four spatial groups of emoji clusters, the football stadium (upper left), the Zoo (below), the botanical garden (upper group) and the Junge Garde (lower right), an outdoor music venue.
Concat emoji based on cluster group/spatial intersection¶
intersects = cluster_sel.sjoin(cluster_sel[["geometry"]], how="left", predicate="intersects").reset_index()
cluster_groups = intersects.dissolve("index_right", aggfunc="min")
Join back the group-id's
cluster_sel["group"] = cluster_groups["index"]
cluster_lists = cluster_sel.groupby("group")["ImpTag"].apply(list)
cluster_lists
Generate images for cluster-groups¶
emoji_cluster_1 = list(cluster_sel[cluster_sel["group"]==1]["ImpTag"])
emoji_cluster_1
Generate sample images for clusters
for ix, cluster_list in enumerate(cluster_lists):
print(cluster_list)
generate_samples(
f"A map icon of happy ({cluster_list[0]}), {''.join(cluster_list[1:])}", save_name=f"emoji_{ix:03d}", rembg=True)