Python data science 101, 12/14/2023, Alexander Dunkel

<span class="smallbold">Lecture & Workshop Series, TU Dresden, December 14, 2023</span>

### <span class="highlight">Python data science 101</span>  
<span class="smallbold highlight">for Social Media Cartography (and beyond)</span>

<span class="smallbold">
Dr.-Ing Alexander Dunkel<br>
Technische Universität Dresden, Germany<br>
Environmental Sciences, Cartographic Communication<br>
</span>

Notes:
- 101 (pronounced ONE-oh-ONE) is a slang term for the most basic knowledge in some subject

---

Web Slides:</br>
<strong><a href="https://tud.link/58yy">https://tud.link/<span class="highlight">58yy</span></a></strong>

---

<p class="highlight">How to use these slides:</p>
<i class="highlight fa fa-chevron-right"></i> Right/Left arrow: Switch slides  
<p> External links <a href="https://ad.vgiscience.org/lbsm-python-lecture/"> <sup class="smallsuper"><i data-customTooltip="Links to external further information, guides etc." class="fa fa-external-link"></i></sup></a></p>
<span class="hoverElement" data-customTooltip="This is a web presentation produced with Jekyll and reveal.js"> Hover to see tooltips </span>
<p>Press <kbd class="hoverElement" data-customTooltip="I mean your s-key, not this button.">S</kbd> to see slide notes.</p>

Notes:
- Broad topic, cannot be covered fully in one lecture
- overview; links to further information available

<div style="height:99vh">
<br>
<strong>Please vote: E-Learning-Schmuckstück 2023</strong>
<img style="height:450px;" class="plain"  src="images/qr_code_schmuckstueck.png"/><br>
<span class="fragment"><strong><a href="https://tud.link/wud6">https://tud.link/<span class="highlight">wud6</span></a></strong></span>
</div>

### Topics

<ol class="roman">
  <li>Data retrieval</li>
  <li>Data structures, formats, attribute mapping</li>
  <li>Types of incoming spatial data</li>
  <li>Data handling, aggregation</li>
  <li>Visualization</li>
  <li>Packages</li>
  <li>Environment setup and tools </li>
  <li>Code conventions</li>
  <li>Tracking: git, revision control</li>
  <li>Reproducibility: docker, WSL, Jupyter</li>
</ol>

Notes:
- structure of the first part (i.-v.) according to common steps
- second part introduction to background tools & concepts

---

<span class="highlight">**Jupyter Lab Workshop**</span>  
Social Media, Data Analysis, & Cartography

- First Workshop
- Geopool HÜL S590
- <i class="hoverElement" data-customTooltip="A few hours are needed until your resources are allocated. Please register ahead of time. The link will be renewed once in a while.">Please register yourself through the link (see email)</i>

Notes:
- please register yourself ahead of time
- From remote: prepare VPN connection to TUD (OpenVPN, Cisco)

## Why Python?

- no vendor-lock in
- mature cartography & data science packages
- interoperability

Notes:
- Example: issues with vendor-lock in when switching workplaces (e.g. ESRI)
- compatibility, ease of integration into existing workflows

---

<div style="width:100%;position:absolute;bottom:100px;text-align:right;z-index:9999;">
<p style="vertical-align: bottom;" >
<strong class="highlight">The goal: </strong></br>
Structured </br>workflows </br>to process data.</br>
</p> 
</div>

Notes:
- Complex workflows: Not an issue, common for most cartographic visualization
- Overall Goal: Structured, reproducible processing pipelines

---

<div style="width:95%;position:absolute;bottom:100px;text-align:right;z-index:9999;">
<p style="vertical-align: bottom;" >
<strong>The reality</strong> <sup><a href="https://pythondatalab.wordpress.com/machine-learning/machine-learning-roadmap/" class="smallsuper"><i data-customTooltip="MACHINE LEARNING ROADMAP" class="fa fa-external-link"></i></a></sup>
</p> 
</div>

Notes:
- In reality, retrieval of data is a time consuming task
- 50% or more project time needed
- increasing when Social Media data is used
- not enough time for the more complicated, second part (data processing)

## Retrieval & Data filtering

Notes:
- following slides: overview of the critical tasks
- Explain common issues and pitfalls during data retrieval

---

<span class="hoverElement" data-customTooltip="Application Programming Interface">API</span>s

- Flickr (flickr.photos.search) <a href="https://www.flickr.com/services/api/flickr.photos.search.html"> <sup class="smallsuper fa fa-external-link"></sup></a>
- Instagram (GraphQL) <a href="https://developers.facebook.com/docs/instagram-api/"> <sup class="smallsuper fa fa-external-link"></sup></a>
- Twitter (streaming API) <a href="https://developer.twitter.com/en/docs/tutorials/consuming-streaming-data"> <sup class="smallsuper fa fa-external-link"></sup></a>

Notes:
- Web APIs comparable, with custom syntax each
- e.g. Twitter: Mostly streaming based (listening necessary)

---

Example: <span class="highlight">Instagram API</span>

```python
hashtag = "park"
query_url = f'https://www.instagram.com/explore/tags/{hashtag}/?__a=1&__d=dis'
```

<ul>
<li class="fragment" data-fragment-index="1"><code class="highlight">?__a=1&__d=dis</code>: return as json</li>
<li class="fragment" data-fragment-index="2"><code class="highlight">f'{var}'</code>: f-strings <sup><a href="https://docs.python.org/3/tutorial/inputoutput.html#formatted-string-literals" class="smallsuper"><i data-customTooltip="> Python 3.6: Formatted String Literals." class="fa fa-external-link"></i></sup></a></li>
<li class="fragment" data-fragment-index="3">Live Query to the API <sup><a href="https://www.instagram.com/explore/tags/park/?__a=1&__d=dis" class="smallsuper"><i data-customTooltip="Works with & without login." class="fa fa-external-link"></i></sup></a></li>
</ul>

Notes:
- Instagram API sample
- Web API, GraphQL, discontinued but still publiy (why?)
- Consumer views endpoint as a url (path)
- API processes requests based on the path/endpoint

---

<pre class="python"><code data-trim data-noescape data-line-numbers="1-10|4|4-6|4-10" data-fragment-index="1">
hashtag = "park"
query_url = f'https://www.instagram.com/explore/tags/{hashtag}/?__a=1&__d=dis'

import json, requests

response = requests.get(url=query_url)

if not response.status_code == 429:
    json_text = response.text
    json_data = json.loads(json_text)
</code></pre>
<ul>
  <li class=fragment data-fragment-index="1"><code class="highlight">requests</code> <sup><a href="https://docs.python-requests.org/en/latest/" class="smallsuper"><i data-customTooltip=">Built for human beings." class="fa fa-external-link"></i></a></sup></li>
  <li class=fragment data-fragment-index="1"><code class="highlight">json</code> Module <sup><a href="https://docs.python.org/3/library/json.html#module-json/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
  <li class=fragment data-fragment-index="3">HTML <code class="highlight">Status Codes</code> <sup><a href="https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

Note:
- python can be used (requests library),
- similar to how the browser interacts with the API
- HTML error codes: 400 Bad requests, 403 Forbidden, 429 Too Many Requests, 500 Internal Server Error, 503 Service Unavailable, 504 Gateway Timeout

---

Frequent <span class="highlight">HTML status codes</span> (e.g.):

| HTML Status Code | Message/Meaning       |
| ---------------- | --------------------- |
| 200              | OK                    |
| 400              | Bad request           |
| 403              | Forbidden             |
| 429              | Too Many Requests     |
| 500              | Internal Server Error |
| 503              | Service Unavailable   |
| 504              | Gateway Timeout       |

</span>

Note:
- ask: how would you react to these codes?

---

<pre class="python"><code data-trim data-noescape data-line-numbers="2|3" data-fragment-index="1">
...
json_data = json.loads(json_text)
json.dumps(json_data, indent=2)[0:550]
</code></pre>

json: <span class="highlight">Flexible, structured, nested</span> data format
```json
{
  "graphql": {
    "hashtag": {
      "id": "17841563668080313",
      "name": "park",
      "allow_following": false,
      "is_following": false,
      "is_top_media_only": false,
      "profile_pic_url": "https://scontent-frt3-1.cdninstagram.com/v/t51.2885-15/e15/c179.0.721.721a/s150x150/166744655_1719090121604264_9033828806473210135_n.jpg?tp=1&_nc_ht=scontent-frt3-1.cdninstagram.com&_nc_cat=108&_nc_ohc=EBg1uEzip80AX_Za9Ag&ccb=7-4&oh=5d812109d80baaa71a671236d9daaa3d&oe=608E3E1B&_nc_sid=4efc9f",
      "edge_hashtag_to_media": {
        "cou ...
```

</span>

<div class="fragment" data-fragment-index="2" style="width:95%;position:relative;bottom:10px;text-align:right;z-index:9999;">
<p style="vertical-align: bottom;" >
Open structure means not very suited for visualization.</p> 
</div>

Notes:
- common return format: json
- attribute–value pairs <code>"attribute":"value"</code>
- array (list) `[{object},{object}]`
- nesting makes it flexible, but problematic for visualization (missing standardization)

---

JSON <i class="highlight fa fa-arrow-right"></i> CSV

Notes:
- common raw interchange format for visualizations: CSV

---

<div style="width:95%;position:absolute;bottom:100px;text-align:right;z-index:9999;">
<p style="vertical-align: bottom;" >
<strong>Flickr CC </br>100 Million </br>Dataset (CSV) </strong> <sup><a href="https://lbsn.vgiscience.org/tutorial/yfcc-introduction/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>
</p> 
</div>

Notes:
- Social Media CSV example: Flickr YFCC 100 Million dataset
- published in 2014 by Yahoo/Flickr, based on CC By images
- same data can be retrieved using the public Flickr API

---

Example: Flickr CC100M CSV

```python
from pathlib import Path

root = Path.cwd()
csv = root / 'data' / 'yfcc100m.csv'
```
<ul>
<li>Define path to CSV</li>
<li class="fragment" data-fragment-index="1"><code class="highlight">pathlib.Path</code>: OS independent filesystem paths <sup><a href="https://docs.python.org/3/library/pathlib.html#basic-use" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="2"><code class="highlight">Path.cwd()</code>: Current directory</li>
<li class="fragment" data-fragment-index="3">Size of dataset: <code class="highlight">~60 GB</code></li>
</ul>

Notes:
- basic code to open the CSV file
- use of pathlib!
- Is it a good idea to open a 60GB file at once?

---

Load CSV
```python
import pandas as pd

csv = "yfcc100m.csv"
df = pd.read_csv(csv, encoding='utf-8', sep='\t', header=None, nrows=5)
```

<ul>
<li class="fragment" data-fragment-index="1"><code class="highlight">Pandas</code> - Python's Excel (understatement) <sup><a href="https://pandas.pydata.org/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="2"><code class="highlight">df</code>? DataFrame? 10 minutes to pandas <sup><a href="https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="3">Docs: <code class="highlight">pd.read_csv()</code> <sup><a href="https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<ul>
<li class="fragment" data-fragment-index="4"><code class="highlight">encoding='utf-8'</code> Character encoding</li>
<li class="fragment" data-fragment-index="5"><code class="highlight">sep='\t'</code> CSV column separator</li>
<li class="fragment" data-fragment-index="6"><code class="highlight">header=None</code> Specify wether first row contains column labels</li>
<li class="fragment" data-fragment-index="7"><code class="highlight">nrows=5</code> Fetch first 5 rows</li>
</ul>
</ul>

Notes:
- pandas to the rescue: Importance of DataFrame in current data science
- streaming of CSV file (chunked processing)
- explain basic syntax

---

Preview
```python
df.head()
```

Notes:
- Easy preview
- immediate observations: Missing header
- Public dataset, information difficult to find about meaning of columns
- common situation; other issues observable

---

Standardization: Attribute Mapping

Notes:  
- conversion JSON to tabular data structure makes standardization necessary
- not a complicated but time intensive task

---

<div style="width:95%;position:relative;top:500px;text-align:right;z-index:9999;">
<p class="highlight" style="vertical-align: bottom;" data-customTooltip="A common language independent, privacy-aware and cross-network social-media data scheme." >
<strong>lbsn.vgiscience.org</strong> <sup><a href="https://lbsn.vgiscience.org/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>
</p> 
</div>

</div>

Notes:
- Integration of data
- time intensive, low-level task
- ability to create visualizations independent of data source

---

YFCC100M: First item.
```python
df.loc[0]
```
<ul>
<li class="fragment" data-fragment-index="1"><code class="highlight">loc-Indexer</code>: Access rows and columns by label(s) or a boolean array. <sup><a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.loc.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

<div class="fragment" data-fragment-index="5" style="width:100%;position:absolute;top:250px;right:0px;text-align:right;z-index:9999;">
<p class="highlight" style="vertical-align: bottom;">
<strong>Flickr </br>API </br>Docs</strong></br> <sup><a href="https://www.flickr.com/services/api/flickr.photos.geo.photosForLocation.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>
</p> 
</div>

Notes:
- explain DataFrame index
- importance of loc indexer compared to other selection methods
- geospatial data peculiarities in Flickr

---

Other methods for deriving spatial information

Gazetteer example: <span class="highlight">Instagram API</span>

Notes:
- What is a Gazetteer?

---

```python
location = "1893214"
query_url = f'https://www.instagram.com/explore/locations/{location}/?__a=1&__d=dis'
```

<ul>
<li>Link to query <sup><a href="https://www.instagram.com/explore/locations/1893214/?__a=1&__d=dis" class="smallsuper"><i class="fa fa-external-link"></i></sup></a></li>
</ul>

<pre><code>
{"graphql":null,"native_location_data":{"location_info":{"location_id":"1893214","facebook_places_id":"119533068103485","name":"Gro\u00dfer 
Garten","phone":"03 51-4456-600","website":"http://www.grosser-garten-dresden.de/","category":"Public 
Square / Plaza","price_range":2,"hours":{"status":"Open 24 hours","current_status":"Open 
24 hours","hours_today":"","schedule":[]},"lat":51.037697,"lng":13.762865,"location_address":"Hauptallee 
10","location_city":"Dresden, Germany","location_region":0,"location_zip":"01219","ig_business":{},"show_location_page_survey":false,"num_guides":null,"has_menu":false,"page_effect_info":{"num_effects":0,"thumbnail_url":null,"effect":null}},"ranked":{"sections":[{"layout_type":"media_grid","layout_content":{"medias":[{"media":{"taken_at":1635240067,"pk":2692911326172298695,"id":"2692911326172298695_1677335629","device_timestamp":163524001910942,"media_type":1,"code":"CVfJJVgID3H","client_cache_key":"MjY5MjkxMTMyNjE3MjI5ODY5NQ==.2","filter_type":0,"is_unified_video":false,"location":{"pk":1893214,"short_name":"Gro\u00dfer 
Garten","facebook_places_id":119533068103485,"external_source":"facebook_places","name":"Gro\u00dfer 
Garten","address":"Hauptallee 10","city":"Dresden, Germany","has_viewer_saved":null,"lng":13.762865,"lat":51.037697,"is_eligible_for_guides":true},"lat":51.037697,"lng":13.762865,"user":{"pk":1677335629,"username":"radio_dresden","full_name":"Radio 
[...]
</code></pre>

</div>

Notes:
- location endpoint
- coupling between Instagram and Facebook Places database
- open in Firefox tab and explain json structure

---

Notes:
- broad overview, more can be done
- amount of work depends on specific context and scrutiny of the analyst

---

<ol class="roman">
  <li class="highlight">Data retrieval</li>
  <li class="highlight">Data structures, formats, attribute mapping</li>
  <li class="highlight">Types of incoming spatial data 
  <li>Data handling, aggregation</li>
  <li>Visualization</li>
  <li>Packages</li>
  <li>Environment setup and tools </li>
  <li>Code conventions</li>
  <li>Tracking: git, revision control</li>
  <li>Reproducibility: docker, WSL</li>
</ol>

<div style="width:100%;position:absolute;top:200px;right:0px;text-align:right;z-index:9999;">
<p class="highlight" style="vertical-align: bottom;">
<strong>First part</strong><sup><i class="fa fa-check"></i></sup>
</p> 
</div>

Notes:
- brief break; ask questions
- Why is the Instagram API still publicly accessible, without authentication?

---

<ol class="roman">
  <li>Data retrieval</li>
  <li>Data structures, formats, attribute mapping</li>
  <li>Types of incoming spatial data 
  <li class="highlight">Data handling, aggregation</li>
  <li class="highlight">Visualization</li>
  <li>Packages</li>
  <li>Environment setup and tools </li>
  <li>Code conventions</li>
  <li>Tracking: git, revision control</li>
  <li>Reproducibility: docker, WSL</li>
</ol>

Aggregation Example: Usercount per Country

Notes:
- introduce simple application example:
- given: users measured per 100km grid cells
- summarize user counts per country (YFCC dataset)

---

Two high-level tasks

1. Assign 100x100 km cells to countries <span class="fragment" data-fragment-index="1">(<span class="highlight">A geometric operation</span>)</span>
2. Count distinct users per country <span class="fragment" data-fragment-index="1">(<span class="highlight">A summary/aggregation operation</span>)</span>

Notes:
- complex task: divide into subtasks

---

Supplementary data <i class="highlight fragment fa fa-arrow-right" data-fragment-index="1"> Integration of information</i>

Notes:
- often, integration of data necessary, 
- based on additional, external information

---

Define origin of supplementary data in your code:
```python
DATA_PATH = Path.cwd() / "data"
FILENAME = "ne_50m_admin_0_map_subunits.shp"
```
<ul>
<li class="fragment" data-fragment-index="1"><code class="highlight">Natural Earth</code> map units shapefile (1:50m) <sup><a href="https://www.naturalearthdata.com/downloads/50m-cultural-vectors/50m-admin-0-details/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span></li>
</ul>

Notes:
- reproducibility also means that origin of data is tracked
- e.g. update/new release

---

Preview data
<pre class="python"><code data-trim data-noescape data-line-numbers="1-2|3|4-6" data-fragment-index="1">
DATA_PATH = Path.cwd() / "data"
FILENAME = "ne_50m_admin_0_map_subunits.shp"
import geopandas as gp
world = gp.read_file(
    DATA_PATH / FILENAME)
world.plot()
</code></pre>

<ul>
<li class="fragment" data-fragment-index="1"><code class="highlight">Geopandas</code>: DataFrame extension for geospatial data<sup><a href="https://geopandas.org/en/stable/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment showmap" data-fragment-index="2"><code class="highlight">matplotlib.pyplot.plot</code><sup><a href="https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.plot.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

Notes:
- fast preview generation of geospatial data
- with geopandas and matplotlib
- Explain geopandas as a spatial extension to pandas
- Access to all methods of pandas, spatial

---

Define <span class="highlight">full</span> origin of supplementary data in your code:
```python
DATA_PATH = Path.cwd() / "data"
URI = "https://www.naturalearthdata.com/http//www." \
      "naturalearthdata.com/download/50m/cultural/"
FILENAME = "ne_50m_admin_0_map_subunits.zip"
```

Notes:
- Improvement: define _full_ origin of data

---

This includes unzipping!

<span class="fragment" data-fragment-index="1">Unzipping? In python? Without writing to disk?</span><span class="fragment highlight" data-fragment-index="2"> <img class="plain highlight" style="width:75px;bottom:-10px" src="assets/grin-stars-solid.svg"/> Google <sup><a href="https://www.google.com/search?q=python+unzipping+a+.zip+file+without+writing+to+disk" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span>

Notes:
- ReadTheDocs or ask Google?
- A single Package: ReadTheDocs
- Broader questions (packages unknown): Google is Ok, but..

---

Notes:
- Python is 20 years old
- all kinds of knowledge from different versions,
- people with different skills
- makes it very difficult and problematic to copy&paste

ChatGPT? <div><img class="fragment plain" style="width:150px;bottom:-50px" src="assets/512.webp"/></div>

---

Four lines of code:
<pre class="python"><code data-trim data-noescape data-line-numbers="1-4|1|2|3|4" data-fragment-index="1">
import requests, zipfile, io
r = requests.get(f'{URI}{FILENAME}')
z = zipfile.ZipFile(io.BytesIO(r.content))
z.extractall(DATA_PATH)
</code></pre>
<ul>
<li class="fragment" data-fragment-index="1"><span class="highlight">zipfile</span>, <span class="highlight">io</span>: Base packages included with Python</li>
<li class="fragment" data-fragment-index="2">Retrieve data</li>
<li class="fragment" data-fragment-index="3">Read response into ZipFile Object</li>
<li class="fragment" data-fragment-index="4">Extract to <span class="highlight">DATA_PATH</span> directory</li>
</ul>

Notes:
- explore options, but consciously write code

---

<div style="height:99vh;position:relative"><br><br><br><br><br>
<span class="fragment fade-out" data-fragment-index="1">Task 2: Counting users</span>
<br>
<div class="fragment" data-fragment-index="1" style="position:absolute;top:0;left:0;height:600px;width:100%;z-index:0">
    <img class="plain" style="width:100%;margin-top:0px;position:absolute;left:50%;transform:translateX(-50%)" src="images/grid_shapes.png"/>
</div>

<div class="highlight fragment r-fit-text" data-fragment-index="1" style="width:100%;height:100px;position:absolute;top:550px;left:50px;text-align:left;z-index:1000;background-color: rgba(255,255,255, 0.8);">
<ul style="vertical-align: bottom;">
<li>Background: Pre-aggregated grid with a set of user for each cell</li>
<li class="fragment" data-fragment-index="2">Foreground: Country shapes</li>
</ul>
</div>
</div>

Notes:
- back to application context
- illustrate starting situation

---

<pre class="python"><code data-trim data-noescape data-line-numbers="1-4|1|2-3|4-5" data-fragment-index="1">
users_set = set()
user = "96117893@N05"
users_set.add(user)
print(len(users_set))
> 1
</code></pre>
Counting distinct users: Low-level approach
<ul style="vertical-align: bottom;">
<li class="fragment" data-fragment-index="1">Define empty set</li>
<li class="fragment" data-fragment-index="2">Add first user-id to set</li>
<li class="fragment" data-fragment-index="3">Calculate the length of the set</li>
</ul>

Notes:
- common approach to divide complex tasks into simple sub-tasks
- simplest way to count distinct users in python

---

Intersection

```python
grid_overlay = gp.overlay(
    grid, world, 
    how='intersection')
```

<ul>
<li><code class="highlight">grid</code> & <code class="highlight">world</code>: GeoDataFrames<sup><a href="https://geopandas.org/en/stable/docs/reference/geodataframe.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span></li>
<li class="fragment" data-fragment-index="1"> GeoDataFrames support all functions of regular Pandas DataFrames (via inheritance)</li>
<li class="fragment" data-fragment-index="2"><code class="highlight">Overlay operations</code>: </br>‘intersection’, ‘union’, ‘identity’, ‘symmetric_difference’ or ‘difference’ <sup><a href="https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.overlay.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span></li>
</ul>

Notes:
- common geometric operations, similar to standard GIS
- explain GeoDataFrame

---

<br>

Preview Intersection
<pre class="python"><code data-trim data-noescape data-line-numbers="1-7|2|3|4-7" data-fragment-index="1">
country_col = 'su_a3' # Column with country codes
query = grid_overlay[country_col].isin(["DEU", "FXX"])
sel = grid_overlay[query]
sel.plot(
    edgecolor='white',
    column=country_col,
    linewidth=0.3)
</code></pre>
<ul>
<li class="fragment" data-fragment-index="1"><code class="highlight">query</code>: Boolean Indexing ("list" of True/False)<sup><a href="https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#basics" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span></li>
<li class="fragment" data-fragment-index="2"><code class="highlight">df[query]</code>: Create a "slice" of the DataFrame</span></li>
</ul>

Notes:
- explain chaining of operations
- preview generation
- possible to split complex tasks further into separate lines

---

```python
grid.plot(edgecolor='white', column=country_col, figsize=(22,28), linewidth=0.3)
```

Notes:
- final result of the intersection

---

Count users:
<pre class="python" data-id="code-animation2"><code data-trim data-noescape data-line-numbers data-fragment-index="1">
usercounts_country = grid["user_guid"].groupby(country_col).size()
</code></pre>
<ul>
<li style="display: none;">Select column for counting (<code class="highlight">pd.Series</code>)<sup><a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li style="display: none;">Group user ids per country column (<code class="highlight">.groupby()</code>)<sup><a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.groupby.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li style="display: none;">Return the number of elements in the underlying data. (<code class="highlight">.size()</code>)<sup><a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.size.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

---

Count users:
<pre class="python" data-id="code-animation2"><code data-trim data-noescape data-line-numbers="1-3|1|2|3" data-fragment-index="1">
usercounts_country = grid["user_guid"] \
    .groupby(country_col) \
    .size()
</code></pre>
<ul>
<li class="fragment" data-fragment-index="1">Select column for counting (<code class="highlight">pd.Series</code>)<sup><a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="2">Group user ids per country column (<code class="highlight">.groupby()</code>)<sup><a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.groupby.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="3">Return the number of elements in the underlying data. (<code class="highlight">.size()</code>)<sup><a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.size.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

Notes:
- Remaining step: Counting/aggregating users per country
- explain pandas operations
- Duplicate counts!

---

Count <span class="highlight">distinct</span> users:
<pre class="python"><code data-trim data-noescape data-line-numbers="1-5|1|2|3|4|5" data-fragment-index="1">
series_grouped = grid["user_guid"] \
    .groupby(country_col) \
    .apply(set)
usercounts_country = series_grouped.apply(len)
world.loc[usercounts_country.index, "usercount"] = usercounts_country
</code></pre>
<ul>
<li class="fragment" data-fragment-index="1">1. Take column "user_guid" </li>
<li class="fragment" data-fragment-index="2">2. Group by "country" column </li>
<li class="fragment" data-fragment-index="3">3. Apply "set" to resulting value-groups </li>
<li class="fragment" data-fragment-index="4">4. Get length of sets (number of distinct users) </li>
<li class="fragment" data-fragment-index="5">5. Store result into new column "usercount"</li>
</ul>

Notes:
- Users may be present multiple times
- count distinct: similar operations
- recapitulate simple low-level user counting with sets

---

Visualize data

---

```python
import geoviews as gv
gv.Polygons(
    world, crs=crs.Mollweide(), vdims='usercount')
```

<ul class="fragment" data-fragment-index="1">
<li class="highlight" style="background-color: rgba(255,255,255, 0.9);">Geoviews<sup style="pointer-events: auto"><a href="https://geoviews.org/index.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>
</div>

Notes: 
- fast preview with matplotlib during development
- sharing with others: Interactive Visualizations
- Geoviews compatible with GeoDataFrames (ad-hoc)

---

<pre class="python"><code data-trim data-noescape>
gv.Polygons(world, crs=crs.Mollweide(), vdims='usercount') + \
    gv.Path(world, crs=crs.Mollweide())
</code></pre>
<br><br>
<ul class="fragment" data-fragment-index="1">
<li class="highlight">Stack or combine plots<sup style="pointer-events: auto"><a href="https://geoviews.org/user_guide/Geometries.html#feature" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>
</div>

Notes:
- explain basic syntax for map layout/layers in gv

---

Export to HTML file
```python
import holoviews as hv
hv.save(layers, 'sample.html', backend='bokeh')
```

<div class="fragment highlight" data-fragment-index="1" style="width:100%;position:absolute;top:350px;left:0px;text-align:left;z-index:9999;">
<p>Workshop </br>result</p>
</div>
</div>

Notes:
- Pyviz: Visualization packages with compatible APIs
- Holoviews: high-level plotting API to Bokeh interactive maps
- example: Generate standalone archive HTML from Geoviews Layers

---

<ul>
  <li ><span class="highlight">Export</span> data to shapefiles <span class="highlight">at any time</span></li>
  <li >& continue layouting in <span class="highlight">QGis</span> (or ArcMap etc.)</li>
  <li class="fragment" data-fragment-index="1"><span class="highlight">Integrate python gradually</span>, by replacing small parts of your workflow</li>
</ul>

Notes:
- Unlike proprietary software, interoperability is a strength
- Incremental and gradual integration of python into existing workflows

---

<ol class="roman">
  <li >Data retrieval</li>
  <li >Data structures, formats, attribute mapping</li>
  <li >Types of incoming spatial data 
  <li class="highlight">Data handling, aggregation</li>
  <li class="highlight">Visualization</li>
  <li>Packages</li>
  <li>Environment setup and tools </li>
  <li>Code conventions</li>
  <li>Tracking: git, revision control</li>
  <li>Reproducibility: docker, WSL, Jupyter</li>
</ol>

<div style="width:100%;position:absolute;top:256px;right:0px;text-align:right;z-index:9999;">
<p class="highlight" style="vertical-align: bottom;">
<strong>Second part</strong><sup><i class="fa fa-check"></i></sup>
</p> 
</div>

Notes:
- Further steps not shown here is extensibility through extensions 
- & customizations of visualization (open ended topic)

<ol class="roman">
  <li>Data retrieval</li>
  <li>Data structures, formats, attribute mapping</li>
  <li>Types of incoming spatial data 
  <li>Data handling, aggregation</li>
  <li>Visualization</li>
  <li class="highlight">Packages</li>
  <li class="highlight">Environment setup and tools </li>
  <li class="highlight">Code conventions</li>
  <li class="highlight">Tracking: git, revision control</li>
  <li class="highlight">Reproducibility: docker, WSL, Jupyter</li>
</ol>

<div style="width:100%;position:absolute;bottom:150px;text-align:right;z-index:9999;">
    <p class="fragment" data-fragment-index="1" style="vertical-align: bottom;" >
        <span class="highlight"><i class="highlight fa fa-arrow-right"></i></span> The 'Chore', </br>Orchestration
    </p>
    <p class="fragment" data-fragment-index="2" style="vertical-align: bottom;" >
        <span class="highlight"><i class="highlight fa fa-arrow-right"></i></span> Main goal: </br><u>Reproducibility</u>
    </p>
</div>

Notes:
- commonly not covered at university
- frequent changes; largely depends on application context
- singular, opinionated perspective

---

Why Reproducibility?

- sharing of knowledge with others
- later reuse of code-parts, transferability
- incremental knowledge construction

Notes:
- reproducibility becomes increasingly important for bigger projects and groups

---

<div style="width:100%;position:absolute;top:350px;text-align:right;z-index:9999;">
    <p class="fragment" data-fragment-index="1" style="vertical-align: bottom;" >
        <span class="highlight">Save your </br>friends from </br>dependency hell </br><sup><a href="https://towardsdatascience.com/devops-for-data-science-making-your-python-project-reproducible-f55646e110fa" class="smallsuper"><i data-customTooltip=">Devops for Data Science: Making your Python Project Reproducible" class="fa fa-external-link"></i></a></sup></span>
    </p>
</div>
</div>

Notes:
- this is where all python developers have been at some point

---

<div style="width:100%;height:100px;position:absolute;top:550px;text-align:center;z-index:1000;" class="fragment" data-fragment-index="1">
<p style="vertical-align: bottom;background-color:white" >
<span class="highlight">An opinionated list of packages for cartography <sup><a href="https://gitlab.vgiscience.de/lbsn/tools/jupyterlab" class="smallsuper"><i data-customTooltip=">IfK Jupyter Lab Container" class="fa fa-external-link"></i></a></sup></span></p> 
</div>
</div>

Notes:
- starts with choice of packages (carto example)
- recommendation: Choose a set of packages that are compatible
- particularly: where developers care for interoperability

---

Environments and package managers (2022)

<ul>
<li><span class="fragment strike" data-fragment-index="1">Anaconda</span> <span class="highlight fragment" data-fragment-index="1"><span>Miniconda</span> for data science <sup><a href="https://docs.conda.io/en/latest/miniconda.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span></li>
<li class="fragment" data-fragment-index="2">Alternative: <span class="highlight">venv <sup><a href="https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup> or Poetry <sup><a href="https://python-poetry.org/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span> & pyproject.toml</li>
</ul>

Note:
- highly controversial topic
- 2021: Conda still _Ok_ for data science
- most promising for future and python package managing overall: Poetry

---

Sample <span class="highlight">environment.yml (conda)</span>
<pre class="yaml"><code data-trim data-noescape data-line-numbers="1-15|1|2-4|5|9|14-15" data-fragment-index="1">
name: sampleenv
channels:
  - conda-forge
  - defaults
dependencies:
  - python>=3.7
  - pip
  - colorcet
  - 'geoviews-core=1.8.1'
  - datashader
  - geopandas
  - seaborn
  - xarray
  - pip:
    - python-hll
</code></pre>

<div style="position:relative;width:100%;text-align:center">
<p class="fragment fade-in-then-out" style="position: absolute;width:100%;text-align:center" data-fragment-index="1">Name of environment</p>
<p class="fragment fade-in-then-out" style="position: absolute;width:100%;text-align:center" data-fragment-index="2">Conda "Channels" to install from</p>
<p class="fragment fade-in-then-out" style="position: absolute;width:100%;text-align:center" data-fragment-index="3">List of packages to install</p>
<p class="fragment fade-in-then-out" style="position: absolute;width:100%;text-align:center" data-fragment-index="4"><span class="highlight">pin</span> package versions</p>
<p class="fragment fade-in-then-out" style="position: absolute;width:100%;text-align:center" data-fragment-index="5">Additional packages installed from <span class="highlight">pypi</span></p>
</div>

Notes:
- Even for conda, many ways exist to handle packages
- Example: Custom conda `.yml` format
- describe syntax

---

Restore environment

Notes:
- Process for restoring a conda environment from a environment.yml

---

Sample <span class="highlight">pyproject.toml</span>
<pre class="toml"><code data-trim data-noescape data-line-numbers="1-29|1-5|7-15|17-21|23-24|26-28" data-fragment-index="1">
[tool.poetry]
name = "sink"
version = "0.2.1"
description = ""
authors = ["mcnesium <git@mcnesium.com>"]

[tool.poetry.dependencies]
python = "^3.9"
fastapi = "^0.73.0"
requests = "^2.26.0"
psycopg2-binary = "^2.9.1"
pandas = "^1.3.3"
uvicorn = "^0.15.0"
Shapely = "^1.7.1"
geohashlite = "^0.3.2"

[tool.poetry.dev-dependencies]
pytest = "^6"
flake8 = "^3.9.2"
black = {version = "^21.9b0", allow-prereleases = true}
pytest-order = "^1.0.0"

[tool.poetry.scripts]
sink = "sink.main:main"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
</code></pre>
<div style="position:relative;width:100%;text-align:center">
<p class="fragment fade-in-then-out" style="position: absolute;width:100%;text-align:center" data-fragment-index="1">General package information</p>
<p class="fragment fade-in-then-out" style="position: absolute;width:100%;text-align:center" data-fragment-index="2">List of packages to install</p>
<p class="fragment fade-in-then-out" style="position: absolute;width:100%;text-align:center" data-fragment-index="3">Additional development dependencies</p>
<p class="fragment fade-in-then-out" style="position: absolute;width:100%;text-align:center" data-fragment-index="4">What to run if invoked from as a script via Command Line Interface (CLI)</p>
<p class="fragment fade-in-then-out" style="position: absolute;width:100%;text-align:center" data-fragment-index="5">Specify that this package is build with Poetry</p>
</div>

---

Code conventions (PEP)

Notes:
- Mistake of ignoring code conventions as a beginner
- Motivations for using code conventions

---

<ul>
<li>Following conventions improves the readability of code</li>
<li class="fragment" data-fragment-index="1">PEP 8 -- Style Guide for Python Code <sup><a href="https://www.python.org/dev/peps/pep-0008/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="2">Use <span class="hoverElement" data-customTooltip="Lint, or a linter, is a static code analysis tool used to flag programming errors, bugs, stylistic errors (etc.).">linters</span>: Pylint <sup><a href="https://github.com/PyCQA/pylint" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>, pycodestyle <sup><a href="https://github.com/PyCQA/pycodestyle" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>, black <sup><a href="https://github.com/psf/black" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>
</ul>

Notes:
- not necessary to follow all conventions always
- be conscious

---

Reproducibility: Comments

---

<pre data-id="code-animation"><code data-trim data-line-numbers>
import requests, zipfile, io
r = requests.get(f'{URI}{FILENAME}')
z = zipfile.ZipFile(io.BytesIO(r.content))
z.extractall(DATA_PATH)
</pre>

Notes:
- Example procedure from earlier

---

<pre data-id="code-animation"><code data-trim data-line-numbers>
import requests, zipfile, io
# Get zip file and extract
r = requests.get(f'{URI}{FILENAME}')
z = zipfile.ZipFile(io.BytesIO(r.content))
z.extractall(DATA_PATH)
</pre>

Notes:
- Add a comment: Increase reproducibility

---

<pre data-id="code-animation"><code data-trim data-line-numbers>
import requests, zipfile, io

def get_zip_extract(
    uri: str, filename: str, data_path: Path):
    """Get Zip file and extract to data_path."""
    r = requests.get(f'{uri}{filename}')
    z = zipfile.ZipFile(io.BytesIO(r.content))
    z.extractall(data_path)

get_zip_extract(URI, FILENAME, DATA_PATH)
</code>
</pre>

<ul>
<li class="fragment" data-fragment-index="1">PEP 257 -- Docstring Conventions <sup><a href="https://www.python.org/dev/peps/pep-0257/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="2"><code class="highlight">uri: str</code> Python type hints <sup><a href="https://docs.python.org/3/library/typing.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="3"><code class="highlight">URI, FILENAME, DATA_PATH</code>: User-Defined module level constants (a naming convention)</li>
</ul>

Notes:
- improvement: Separate "business" logic from actual operation
- reusability of code: reduce duplication
- docstrings, type-hints: Powerful feature,
- used e.g. to in automatically generate APIs and documentation (Fastapi)

---

Tools:
<ul>
<li class="fragment" data-fragment-index="1"><span class="fragment strike" data-fragment-index="2">Visual Studio</span>, <span class="fragment fade-in strike" data-fragment-index="3">PyCharm</span><span class="fragment" data-fragment-index="4">, </br><span class="highlight">VSCode/ VSCodium</span> - Standalone code development <sup><a href="https://code.visualstudio.com/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span></li>
<li class="fragment" data-fragment-index="5"><span class="highlight">Jupyter Lab </span> - Combine comments and code <sup><a href="https://jupyter.org/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

Notes:
- 2021 summary of the most important tools for coding

---

<div style="width:100%;position:absolute;bottom:50px;text-align:center;z-index:9999;">
<p style="vertical-align: bottom;background-color:white" >
VSCode started from WSL</p> 
</div>
</div>

Notes:
- Common convention for Windows developers: Frontend through Web/Windows, Backend: Linux
- Why Linux? Reproducibility.
- Example: VSCode started in a folder through WSL/Linux

---

<div style="width:100%;position:absolute;bottom:50px;text-align:center;z-index:9999;">
<p class="highlight" style="vertical-align: bottom;background-color:white" >
Jupyter Lab (Workshop)</p> 
</div>
</div>

Notes:
- Jupyter recommended for beginners

---

Notes:
- combine code & comments; process oriented workflow
- suited for gradual code and visualization development

---

Faster Development of Visualizations

Notes:
- Example changing color of grid
- immediately see visual effect of changes to code

---

IPython makes it easy to <span class="highlight">inspect</span> variables

Notes:
- (use ALT+Left Mouseclick to zoom into small code)
- example of anti-pattern to use print() for variable inspection
- in jupyter, use of `?variable` for inspection
- IPython: Runtime environment oriented

---

Parametrized Visualizations

<div style="width:100%;position:absolute;bottom:50px;text-align:center;z-index:9999;" class="fragment" data-fragment-index="1">
<p style="vertical-align: bottom;background-color:white" >
(Google) Best practices: <br><span class="highlight">From Jupyter notebooks to production ready code<sup><a href="https://cloud.google.com/blog/products/ai-machine-learning/best-practices-that-can-improve-the-life-of-any-developer-using-jupyter-notebooks" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></p> 
</span></div>

Notes:
- (use ALT+Left Mouseclick to zoom into small code)
- Conventions for writing notebooks (Example: Google best Practice)
- Example: Parameterized Notebooks (50km to 500km grid)
- specify parameters, to be changed by users, at the beginning
- prepare code accordingly

---

Direct Export to HTML to share results

Notes:
- Interoperability
- Archive & Sharing of Code, Comments & Output in a single format

---

<span class="highlight">Benefits in summary:</span>
- Reproducibility of the analytical process
- Reusability of code
- Code that is commented and directly connected to generated visuals
- Remote execution of code, local development in browser

---

Not covered so far:<br>
<span class="hoverElement" data-customTooltip="Lightweight, standalone container as an executable package of software that includes everything needed to run an application">Docker</span>, <span class="hoverElement" data-customTooltip="Windows Subsystem for Linux">WSL</span> <sup><a href="https://ad.vgiscience.org/links/posts/2019-05-27-setup-wsl-win10/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>, <span class="hoverElement" data-customTooltip="Code revision tracking">Git</span> <sup><a href="https://ad.vgiscience.org/links/posts/2019-05-27-wsl-software/#git" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>?  
<div class="fragment"><img class="plain highlight" style="width:175px;bottom:-10px;opacity: 1.0;" src="assets/grimace-regular.svg"/></div>

Notes:
- there is more
- not necessary to start with everything;
- take small steps, integrate gradually

---

<span class="highlight">IfK "Carto-Lab" Docker Container</span> <sup><a href="https://gitlab.vgiscience.de/lbsn/tools/jupyterlab" class="smallsuper"><i class="fa fa-external-link"></i></a></sup> & <span class="highlight">Workshop Notebooks</span> <sup><a href="https://gitlab.hrz.tu-chemnitz.de/tud-ifk/python_datascience_2022" class="smallsuper"><i class="fa fa-external-link"></i></a>  </sup>
- Environments prepared with conda
- Notebooks accessed through web browser
- Each notebook is started on a separate machine (ZIH)
- Notebook revisions tracked in git

Notes:
- For cartographers: Have a look at our Jupyter Workflow
- Can be started at home

---

See instructions to run the workshop <span class="highlight">at home</span> <sup><a href="https://gitlab.hrz.tu-chemnitz.de/tud-ifk/python_datascience_2022/-/blob/main/resources/faq/04_offline_setup.md" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>