Mobile Web Applications

1 Overview

Introduction and course organization

1.1 Who am I

• Short CV:
  • Bachelor Audiovisual Media -> worked at a visual effects company
  • Master Computer Science & Media -> co-founded a startup
  • Now: Scientific assistant @HDM and PhD student @UniTübingen
• Lectures:
  • 119640 Mobile Web Applications
  • 113466 Cloud Services and Internet of Things
  • 119310 Mobile App Development 2 (supportive)
• Topics:
  • Full-Stack Web Development (UX/UI, Frontend, Backend, Deployment)
  • Cloud Services and distributed systems
  • Digital accessibility (especially real-time captioning)

1.2 Objective

As a team you build

• A mobile first single-page web application frontend
• A web application backend
• An API to connect frontend and backend
• A toolchain to setup and test your full application

in a technology of your choice

The idea and feature-list of your app does not matter so much, we want to learn workflow and technology here!

1.3 App Ideas

You don't have to build something with a real business case.
You don't have to solve complex problems (e.g. you can fake your AI driven fancy algorithm).
You can create something funny instead.

Here are some ideas to find an application you want to build (no rights reserved):

• 💬 ChatGG: Whatsapp clone to chat with different digital personalitites (e.g. Supportive-Bot, Hate-Bot, Useless-Facts-Bot, Ghosting-Bot)
• 🤑 SplitRight: Share expenses unequally regarding your account balance (e.g. capitalist, communist, swabian, godfather or random mode)
• 🖼 FML (FakeMyLife): Instagram but images are AI-generated from text (e.g. Stable Diffusion)
• 🔓 Weakly: Insecure password manager that generates the weakest passwords satisfying given constraints
• 💞 Lovefinderrz: Dating app based on your favorite color and programming language
• 🗺️ WhereSunny?: Choose your favorite weather and find the closest location
• 🚂 Murphy's Travel: Show train connections with delay estimations and worst case scenarios
• 🍔 FoodYouRather: Find your perfect meal by comparing two dishes multiple times
• 💌 Mailicious: Send scheduled spam emails automatically and at random intervals
• 📉 Broke(r): Forecast which stocks burn your money efficiently
• 📚 Randolingo: Translate words randomly and guess the language afterwards
• 🎲 BoredGames: Classical board games collection

Depending on your idea, you might have additional challenges: charts, animations, audio, video, third-party apis, web-scraping, ...

If you need some extra motivation, you can additionally present your app at the media night. No extra points or ECTS, but the possibility to present it to a larger audience. If you're interested, talk to me.

1.4 Prerequisites

For HdM-students (MI7, MM7):

• 113114 Web Development

These lectures are also helpful:

• 133211 Web Development Frontend
• 113418 Web Development Backend

In general:

• You should have basic experience in developing websites in HTML, JavaScript and CSS.
• Knowledge about gitlab, docker, databases, IT-security is helpful.
• It is expected that you do some coding here (only design or project management is not enough)

1.5 Links

Hybrid?

The course takes place in presence.

If you want to participate remotely (e.g. due to covid infection, quarantine or other legitimate reasons), write me an email early enough. I'll ensure to bring the Meeting Owl and start the BBB-Stream. It's not a high-quality hybrid setup though, just sound and a shared screen.

1.6 Schedule

Lecture Sessions

• We'll talk through this slides in presentations
• Duration is approximately 5 sessions

Team Meetings

• Every team has a fixed 10 minutes slot
• Prepare the meeting (what have you done, what are you planning, specific questions to problems, ...)
• Unexcused absence will make you fail the course
• You can use the remaining lecture time to work on the applications

Working Sessions

• You can use the lecture time to work on the applications
• You'll get support, feedback, ...
• We can solve problems together

Presentation Sessions

• No slides - Show your code and app
• Each member has to present equally
• Unexcused absence will make you fail the course
• Duration: 12 min per team
• Midterm presentations
  • Introduce yourself (who are you, what are you studying, what's your background)
  • Each team presents its current state
  • Check the grading slide for general presentation rules
  • Content: app idea, technology-stack, (maybe schedule, mockups, frontend, backend, docker & code)
  • Keep about two minutes for questions and feedback
• Final presentations
  • Each team presents its final state
  • Check the grading slide for important things to present
  • Open and present live in browser

1.7 App Submission

Grading is based on the Gitlab repo in the mwa group https://gitlab.mi.hdm-stuttgart.de/groups/mwa

Add a README.md in the repo that contains:

• project name
• members (full name, student short, matriculation number)
• short project abstract (only one or two sentences)
• getting started guide
  • explains how to start (should be docker compose up)
  • where to open (e.g. http://localhost:3000)
  • how to login (credentials)
  • and additional requirements if necessary (e.g. how to generate demo data / populate the database)
• testing
  • which components you wrote tests for (frontend and backend)

1.8 Grading

• General

  • The total of 50 Points is split into 5 categories.
  • Following general best practices is required for each category.
  • All members have to contribute with code (only Design or Project Management is not enough)
  • All members have to be present with an adequate amount of commits (pair programming is no excuse, switch roles)
  • Individual grades can be adjusted particularly if the contribution is uneven across the group

• Presentation (10 Points)
  Not all points are relevant for the midterm presentations

  • Each team member has equal presentation time and content
  • Presentation is finished in time (12 minutes)
  • Presentation is well structured, prepared and fluently presented
  • Interesting/unique parts of the app, code, tooling are presented
  • Group highlights lessons learned

• Organization & Tooling (10 Points)

  • Team Meetings are prepared and attended on time
  • Group sticks to aggreements
  • README contains App Submission requirements
  • Group uses gitlab issues, feature branches and merge requests to organize team work
  • After clone, application starts with docker compose up
  • Database is automatically populated with useful demo data
  • Custom dockerfiles are used and serve production builds (backend and frontend)
  • Databases use public docker images
  • Tests are successfully executed by CI-pipeline when pushed to repository

• UX and Design (10 Points)

  • A user understands what to do
  • Application is responsive and works on a phone, tablet, desktop
  • Design is modern and aesthetic (contrast, spacing, color-palettes, ui-elements, ...)
  • Design serves the purpose (e.g. business, game, ...)

• Implementation (20 Points)

  • Application works (no crashes)
  • File and folder structure is clean, makes sense and consistent
  • Code is well readable, consistently formatted and extended with comments where helpful
  • Application implements a form of authentication and authorization (can be a login with a predefined user)
  • Application implements full CRUD functionality on at least two entities (auth endpoints are excluded)
  • Basic backend security is assured (e.g. cors, auth, input validation, password hashing, error-handling, ...)
  • Backend contains unit tests of at least one entity with full service CRUD operations and full test coverage
  • Backend contains end-to-end tests of at least one entity with full CRUD operations and middlewares
  • Frontend contains at least two fully tested components
  • Frontend contains at least one useful end-to-end test

1.9 Ask for help!

In general:

The Center for Learning and Development, Central study guidance, VS aka student government support you:

• Exam nerves, fear of failure, financial problems, stress, depression, ...
• Bullying, racism, sexism, discrimination, ...
• Tipps and feedback regarding scientific writing (e.g. bachelor thesis)
• Career options after the bachelor
• Support for decision-making

Regarding this course:

• Don't be afraid to ask questions about your project (that won't affect your grading negatively)
• Talk to me early if there are any problems within the group (someone never shows up or does not support the group)

1.10 Questions

• Do you know what you will build?
• Do you know how it is graded?
• Do you know what presentation, lecture, working, Q&A sessions etc. are?
• Anything else?

Something funny: Interview with Senior JS Developer in 2022

2 Introduction

Mobile Web Applications

2.1 Web Sites

• Traditional approach to the web
• Server delivers complete websites as a documents
• Each document is identified by a unique URL
• Client renders each document once
• Multi Page Application (MPA)

All application logic is implemented on the server

2.2 Web Applications

• Modern approach to the web
• Server delivers only data using an API
• The whole application is often identified by a single URL
• Client renders the application repeatedly

All application logic is implemented on the client

2.3 Web Sites with Dynamic Parts

• Mixed approach between web sites and web applications
• Server delivers both documents and data using an API
• Each document is identified by a unique URL
• Client renders each document once, but also renders dynamic parts repeatedly

The application logic is implemented both on the server and the client

2.4 Single-Page Applications (SPAs)

• All pages and sub-pages use the same document, e.g. index.html and application
• The application is responsible for rendering different components for different URLs
• This is often accomplished by using an internal router that watches location.href changes

2.5 Meta-frameworks

Meta-frameworks like Next.js, SvelteKit, Nuxt combine server and client and offer various rendering techniques:

• Server Side Rendering (SSR)
• Client Side Rendering (CSR)
• Static Site Generation (SSG)
• Incremental Static Regeneration (ISR)
• Universal Rendering: Hydration converts static HTML into a dynamic web page

Caution: Frameworks might handle these techniques differently (e.g. Next.js ISR and Nuxt ISR)

NuxtJS Hybrid Rendering example

export default defineNuxtConfig({
  routeRules: {
    // Homepage pre-rendered at build time
    "/": { prerender: true },

    // Product page generated on-demand, revalidates in background
    "/products/**": { swr: true }, // swr = stale-while-revalidate

    // Blog post generated on-demand once until next deploy
    "/blog/**": { isr: true }, // isr = incremental static regeneration

    // Admin dashboard renders only on client-side
    "/admin/**": { ssr: false }, // ssr = Server Side Rendering
  },
});

2.6 Progressive Web Applications (PWAs)

• Normal Web Applications with added features
• Usually the goal is to enable features that only native features would offer
• MDN Progressive Web App Docs

Features

2.7 Responsive Web Design (RWD)

• Design approach to make web sites and applications render well on multiple screen sizes
• Often this involves the use of the CSS Flexbox, Grid, Media Queries

Mobile-First Approach

• The mobile first approach starts by designing for the smallest screen first
• Later, the more advanced versions are added
• This process is called progressive advancement

2.8 What do we build in this course?

• In this course we build a Website with Dynamic Parts or a Web Application
• It can be a Single-Page Application, but it does not have to
• It can be a Progressive Web Application, but it does not have to
• We also use a mobile-first approach with a focus on mobile phones

Any Questions?

3 Frontend

Web application running in the browser

3.1 History of Frontend Frameworks

3.2 Document Object Model (DOM)

• Specified by W3C Document Object Model Core
• Describes the elements of the document as objects
• Objects have properties that can be read and set using JavaScript
• Allows HTML access through JavaScript
• All Frameworks (React, Vue, Angular, ...) use this API under the hood

Browser Examples

// select elements
document.querySelector("h2");
document.querySelectorAll("h2");

// remove element
document.querySelector("h2").remove();

// change content
document.querySelector("h2").innerHTML = "hello";

// change style
document.querySelector("h2").style.color = "blue";

// modify class list
document.querySelector("h2").classList.add("active");
document.querySelector("h2").classList.remove("active");

3.3 Plain JS

• Using the DOM and some JavaScript APIs such as fetch, we can already build a web application
• No frameworks, just plain JavaScript
• Does not need to be transpiled
• Extremely lightweight
• Works probably better than you think

Example: Shoppy-plainjs

Is plain JavaScript practical and fast?

3.4 Web Components

• A native approach that is implemented in all major browsers
• No frameworks, support comes out of the box
• MDN Web Components
• Libraries to build Web Components more easily: Lit, stencil

Further Topics

• Custom Elements define your own HTML tags, ...
• Shadow DOM encapsulate components, ...
• Templates and Slots reuse styles, create hierarchy, ...

3.5 Frontend Frameworks

React

• Widely used framework open-sourced by Facebook
• Uses reactive application pattern
• Can be used with or without transpiling
• Uses Virtual DOM and JSX
• Application are often setup with create-react-app
• Next.js evolves into the default react setup

Example: Shoppy-React

Further Topics

• Function and Class Components traditional and new approach
• Hooks useState, useContext, useReducer, ...
• Redux a full scale state management solution

Vue

• Widely used open-source framework
• Uses reactive application pattern with two-way data binding
• Can be used with or without transpiling
• Uses Virtual DOM
• Application are often setup with Vue CLI
• There is a fantastic guide

Example: Shoppy-Vue

Further Topics

• Application & Component Instances Root, Lifecycle Methods, ...
• Template Syntax Interpolations, Directives, Shortcuts, ...
• Pinia a full scale state management solution (Successor of vuex)

Angular

• Widely used framework open-sourced by Google
• Uses reactive application pattern with two-way data binding
• Typescript only
• Uses Incremental DOM
• "Enterprise" framework

Example: Shoppy-Angular

Further Topics

• Tour of heroes Components, Services, Routing, ...
• NGRX a full scale state management solution
• Signals Huge performance rewrite in version 17 (2023) through signals

Svelte

• A pretty new open-source framework
• Uses a new approach and does all work in the compile step
• Still uses a reactive application pattern that is easy to understand
• Exported result is a simple bundle.js and global.css
• Application are often setup with npx degit sveltejs/template my-svelte-project
• There is a brilliant official tutorial

Example: Shoppy-Svelte

Further Topics

• Template Syntax Tags, Attributes, ...
• Stores writeable, readable, get, ...
• Motion Transitions, Animations, ...
• SvelteKIT Similiar to NextJS but for svelte

Further Frameworks

• There are many more frameworks out there
• You can try and use any framework you want to learn
  • Ember
  • Meteor
  • NextJS
  • NuxtJS
  • SolidJS
  • Qwik
  • ...

3.6 Button Counter (Component Example)

JS + HTML (VanillaJS)

<button id="button" onclick="countUp()"></button>

<script>
  let count = -1;
  const countUp = () => {
    count++;
    const element = document.getElementById("button");
    element.innerText = `You clicked me ${count} times.`;
  };

  countUp();
</script>

Vue

<template>
  <button v-on:click="countUp()">You clicked me {{ count }} times.</button>
</template>

<script>
  export default {
    data: function () {
      return {
        count: 0,
      };
    },
    methods: {
      countUp: function () {
        this.count++;
      },
    },
  };
</script>

React

import React, { useState } from "react";

function Counter() {
  const [count, setCount] = useState(0);

  const countUp = () => {
    setCount((prevCount) => prevCount + 1);
  };

  return <button onClick={countUp}>You clicked me {count} times.</button>;
}

export default Counter;

Angular

import { Component } from "@angular/core";

@Component({
  selector: "app-button-counter",
  template: `
    <button (click)="countUp()">You clicked me {{ count }} times.</button>
  `,
})
export class ButtonCounterComponent {
  public count: number = 0;

  countUp() {
    this.count++;
  }
}

Svelte

<script>
  let count = 0;

  function handleClick() {
    count += 1;
  }
</script>

<button on:click="{handleClick}">You clicked me {count} times.</button>

3.7 Transpiling and Bundling

• Browsers only understand JavaScript
• Different browsers support different features
• Developers want to use language extensions such as JSX or Typescript
• Developers also want to use the newest EcmaScript (JS) features, but need backwards compatibility for older browsers.
• Complex applications need to be bundled into single, optimized files
• Babel, SWC are compilers to get browser compatible JavaScript
• Webpack, esbuild, Rollup, Parcel are application bundlers
• Vite Development environment, rollup bundle configuration, HMR (Hot Module Replacement), ...

3.8 Typescript

• Superset of JS: "JavaScript with syntax for types"
• Write strongly typed JS and compile to plain JS for execution
• Reduces errors and enables autocompletion
• Use modern language features and compile to different targets (ES5, ES6, ...)
• Many open-source projects migrated to typescript

// Primitives
const enabled: boolean = true;
let anyValue; // untyped variables will receive "any" type
let anotherValue: any;

// Enums
enum AuthState {
  LOGGED_IN = "logged-in",
  LOGGED_OUT = "logged-out",
}

// Interfaces
interface User {
  name: string;
  age: number;
}

// Typed Arrays
const users: User[] = [{ name: "Jane Doe", age: true }]; // will show error for age value

// Classes
class MyComponent {
  constructor() {}

  render(): string {
    return `<div>My Component</div>`;
  }

  async fetch(): Promise<User | null> {
    // ...
  }
}

// Functions with Generics
const clone = <Type>(value: Type): Type => JSON.parse(JSON.stringify(value));
const clonedUser = clone(user); // clonedUser will be of type User

3.9 Cross-Origin Resource Sharing (CORS)

• Security feature implemented in browsers
• Read and understand: Mozilla CORS

If your frontend and backend run on different origins, your backend needs to whitelist allowed frontend origins

Example

• Frontend runs on localhost:8080
• Backend runs on localhost:8081

Backend needs to allow access from the frontend using a Access-Control-Allow-Origin: http://localhost:8080 header

3.10 Asynchronous programming

• Javascript is single-threaded
• Synchronous task block the event loop
• Many APIs are asynchronous (Http-Requests, Filesystem, Database-Queries, ...)

Node.js event loop

*Source

JavaScript Async Visualizer: Nice tool to try out the call order of async tasks

Callbacks

const findProductsByName = (name, callback) => {
  // Fetch-API has no Callback implementation, so this is just a fake
  fetchWithCallbacks.get("/products", (err, response) => {
    if (err) {
      callback(err);
    } else {
      response.json((err2, data) => {
        if (err2) {
          callback(err2);
        } else {
          const products = data.filter((o) => o.name === name);
          callback(null, products);
        }
      });
    }
  });
};

findProductsByName("Melon", (err, products) => {
  if (err) {
    console.error(err);
  } else {
    console.log(products);
  }
});

Promises

const findProductsByName = (name) => {
  return fetch
    .get("/products")
    .then((response) => {
      return response.json();
    })
    .then((data) => {
      const products = data.filter((o) => o.name === name);
      return products;
    });
};

findProductsByName("Melon")
  .then((products) => console.log(products))
  .catch((err) => console.error(err));

async/await

const findProductsByName = async (name) => {
  const response = await fetch.get("/products");
  const products = await response.json();
  return products.filter((o) => o.name === name);
};

findProductsByName("Melon")
  .then((products) => console.log(products))
  .catch((err) => console.error(err));

Observables & rxjs

RxJS is a library for reactive programming using Observables, to make it easier to compose asynchronous [...] code.

Source: https://rxjs.dev

Example Task:

• Send asynchronous search request to API when input field value changes
• Send request automatically after 1000ms without new input
• Only send request, if value is different from last search request
• Disable input field while request is pending
• Log results to console and enable input field again

Example: rxjs-search

import {
  map,
  fromEvent,
  pluck,
  filter,
  debounceTime,
  distinctUntilChanged,
  tap,
  switchMap,
} from "rxjs";

const products = ["Apples", "Oranges"];
const searchInput = document.getElementById("searchInput") as HTMLInputElement;

const findProducts = (search: string) => {
  return new Promise((resolve) => {
    setTimeout(() => {
      const results = products.filter((o) =>
        o.toLowerCase().includes(search.toLowerCase())
      );
      resolve(results);
    }, 1000);
  });
};

fromEvent(searchInput, "input")
  .pipe(
    pluck("target", "value"), // Use event.target.value as value
    tap((value) => console.log(`input: ${value}`)), // Log value
    map((value: string) => value.trim()), // Remove leading and trailing whitespaces
    filter((value) => value !== ""), // Ignore empty strings
    debounceTime(1000), // Wait 1000ms without changes
    distinctUntilChanged(), // Ignore if value hasn't changed
    tap(() => (searchInput.disabled = true)), // Disable input field
    switchMap((value) => findProducts(value)) // Trigger API-Request
  )
  .subscribe((result) => {
    searchInput.disabled = false;
    console.log(`result: ${JSON.stringify(result)}`);
  });

4 API

Communication layer between frontend and backend

4.1 HTTP Protocol

• HTTP 1
  • is a text based protocol (we can read and understand it)
  • Requests and Responses are split into a header and a body part
  • HTTP 1.1 is defined in RFC2616
• HTTP 2
  • is binary (we can not read and understand it)
  • is defined in RFC7540
• HTTP 3 (alias QUIC)
  • is in development
  • is based on UDP
  • handles encrytion (TLS) inside the protocol

Example Connection

Request

openssl s_client -connect 141.62.64.38:443
GET / HTTP/1.1
Host: mwa.pages.mi.hdm-stuttgart.de

Response

HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=600
Content-Length: 5389
Content-Type: text/html; charset=utf-8
Expires: Wed, 11 Oct 2017 11:05:55 CEST
Last-Modified: Tue, 10 Oct 2017 16:08:50 GMT
Vary: Origin
Date: Wed, 11 Oct 2017 08:55:55 GMT

<!DOCTYPE html>
...

4.2 WebSocket Protocol

• Enables two-way communication between a client and a host
• Data frames can be unicode or binary
• WebSockets are standardized in RFC6455

Example

• MDN Writing WebSocket client applications

4.3 Representational State Transfer (REST)

• Specified by Roy Fielding in his PhD Dissertation, 2000

• Everything is a resource that has a current state

• HTTP methods can be used to transfer resources between states

• Resources are identified by their URL

• Collections are in plural form: users, messages, notifications, ...

• Entities are specified by their collection and id: users/5, messages/y8hd7dg3, notifications, ...

• QueryParameters are optional and typically only used on GET endpoints e.g. for pagination or filtering

• Resources can also be nested, e.g. http://api.example.com/users/5/messages is the collection of messages for user 5

Examples

# List all users
curl -X GET http://api.example.com/users

# List only users matching a searchTerm, sort them by the property name
curl -X GET http://api.example.com/users?searchTerm=Jane&orderBy=name

# Limit results to 50 to provide pagination through large datasets
curl -X GET http://api.example.com/users?page=0&limit=50

Good examples for REST APIs

• Stripe API
• Trello API
• Dropbox API
• Github API
• SoundCloud API

4.4 GraphQL

• GraphQL is a query language for APIs and a runtime for fulfilling queries with data
• It was developed by Facebook in 2012 and open-sourced in 2015
• Specification and guides at graphql.org
• GraphiQL is a very helpfull web app that helps you interacting with a GraphQL API

Example

• Official GraphiQL Demo

5 Backend

Web application running on a server

5.1 Static File

• Simple, read-only form of a backend
• Combined with an automated Git/CI deploy quite powerful

[
  { "id": 1, "name": "Apple", "price": 0.2 },
  { "id": 2, "name": "Banana", "price": 0.8 },
  { "id": 3, "name": "Melon", "price": 1.2 },
];

5.2 Node.js

• Very popular open-source runtime for JavaScript
• Node.js API Documentation

Example: Shoppy-nodejs

Example

• A simple REST server serving products at the GET /products endpoint

const http = require("http");

// this could be loaded from a database
const products = [
  { id: 1, name: "Apple", price: 0.2 },
  { id: 2, name: "Banana", price: 0.8 },
  { id: 3, name: "Melon", price: 1.2 },
];

// create a server
const server = http.createServer(function (request, response) {
  // always serve json
  response.setHeader("Content-Type", "application/json");

  // primitive route check
  if (request.method === "GET" && request.url === "/products") {
    response.end(JSON.stringify(products));
    return;
  }

  // if we are here, we don't know what to do
  response.writeHead(404);
  response.end();
});

console.log("listening on localhost:8081");
server.listen(8081, "localhost");

5.3 express

• Very popular open-source framework on top of NodeJS-Http
• express Documentation

Example: Shoppy-express

Example

const express = require("express");

// this could be loaded from a database
const products = [
  { id: 1, name: "Apple", price: 0.2 },
  { id: 2, name: "Banana", price: 0.8 },
  { id: 3, name: "Melon", price: 1.2 },
];

// create a server
const app = express();

// define endpoint
app.get("/products", (req, res) => {
  res.json(products);
});

app.listen(3000, () => {
  console.log(`Listening on http://localhost:3000`);
});

5.4 NestJS

• Very popular and trending open-source framework based on NodeJS
• Typescript first
• Abstraction on top of (express or fastify) using patterns from JavaSpring and Angular
• Excellent documentation for many use-cases: authentication, file-upload, websockets, testing ...

Example: Shoppy-nestjs

Example

Controller

import { Controller, Get, Post } from "@nestjs/common";
import { Product, CreateProductDto } from "./product.interfaces.ts";
import { ProductService } from "./product.service.ts";

@Controller("products")
export class ProductController {
  constructor(private productService: ProductService) {}

  @Get()
  findAll(): Product[] {
    return this.productService.findAll();
  }

  @Post()
  create(@Body() createProductDto: CreateProductDto): Product {
    return this.productService.create(createProductDto);
  }
}

Service

import { Injectable } from "@nestjs/common";
import { Product, CreateProductDto } from "./product.interfaces.ts";

@Injectable()
export class ProductService {
  private readonly products: Product[] = [];

  findAll(): Product[] {
    return this.products;
  }

  create(createProductDto: CreateProductDto): Product {
    const id = Math.max(...this.products.map((o) => o.id)) + 1;
    const product = { ...createProductDto, id };
    this.products.push(product);
    return product;
  }
}

5.5 Go

• A popular open-source runtime and language developed by Google
• Statically typed to reduce runtime errors, e.g. TypeScript
• Fantastic standard library enables to build web backends without frameworks
• Server is compiled into a single binary (+-6 MB) that can easily by deployed

Example

• A simple REST server serving products at the GET /products endpoint

package main

import (
  "encoding/json"
  "fmt"
  "log"
  "net/http"
)

// struct for a product
type product struct {
  Name  string
  Price float32
}

// this should be loaded from a database
var products = map[string]product{
  "apple": {Name: "Apple", Price: 0.2, },
  "banana": { Name: "Banana", Price: 0.8, },
  "melon": { Name: "Melon", Price: 1.2,  },
}

func main() {
  // handle http at path /products
  http.HandleFunc("/products", func(w http.ResponseWriter, r *http.Request) {
    // we respond with json
    w.Header().Add("Content-Type", "application/json")

    // lets encode our products in json
    err := json.NewEncoder(w).Encode(products)
    if err != nil {
      http.Error(w, err.Error(), 500)
      return
    }
  })

  fmt.Println("listening on localhost:8081")
  log.Fatal(http.ListenAndServe("localhost:8081", nil))
}

5.6 Further Frameworks

• There are many more frameworks out there
• You can try and use any framework you want to learn

Examples

• JavaScript
  • Sails
  • Feathers
  • Hapi
• Go
  • Gin
  • Buffalo
  • Revel
• Ruby
  • Rails
  • Sinatra
• Python
  • Django
  • Flask
  • FastAPI
• Java
  • Spring Boot
• PHP
  • Laravel
• C++, .NET, C#, ...

5.7 Databases

• There are lots of different types of Databases
• Every type solves a special problem well

There are also client side (in browser) databases:

• Key/Value localStorage, sessionStorage
• Relational IndexedDB

There are powerful libraries that extend the browser storage with database apis: PouchDB, rxdb

5.8 Database API Mappers

• Directly serve data from a database via API
• No extra code - the data structure dictates API structure
• Can be extremely useful and fast to implement

Example

• Hasura maps a PostgreSQL database directly to a GraphQL API
• PostgREST maps a PostgreSQL database directly to a REST API

6 Security

Auth, input validation, output serialization, error handling

6.1 Authentication

• HTTP is stateless, so WHO is making the HTTP request?
• By creating a session on the server, the server can remember the user

Conceptual Flow

1. User with id = 3 sends credentials username and password to an API endpoint
2. API action checks if credentials are correct for the user
3. API creates a session: id = "someRandom123String", user_id = 3
4. The session.id (also calle token) is returned to the user
5. For each future request, the client adds the session.id for authentication

Token Storage

• A client needs to remember the token for authentication
• Cookies, its recommended to use HttpOnly, Secure (see MDN Cookies)
• WebStorage APIs, not as save because there is access from JS (see MDN WebStorage API)

6.2 Authorization

Once we know WHO the user is, WHAT is she allowed to do?

Access Control

• Role-based access control (RBAC), e.g. user is an admin, a reader or writer, ...
• Attribute-based access control (ABAC), e.g. if user.country == "DE"
• Access-control list (ACL), e.g. Alice: read,write and Bob: read

6.3 JSON Web Tokens (JWT)

Excellent explanation and live-testing: jwt.io

• Client: sends credentials (username + password)
• Server: verifies credentials, creates a token, signs its content with a secret key and sends the token to the client
• Client: sends token with every request
• Server: validates token content and signature
• Client manipulation of the tokens content would result in an invalid signature

Encoded JWT (Token)

  eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

Schema:

  header.payload.signature

  header: base64encode(stringify(json))
  payload: base64encode(stringify(json))
  signature: hmacSha256(header + payload, secret)

Decoded JWT

// header
{
"alg": "HS256",
"typ": "JWT"
}

// payload
{
"sub": "1234567890",
"name": "John Doe",
"iat": 1516239022 // issuedAt
}

Example

Of course there are robust and secure libraries, this is just an example:

function createJWT() {
  const header = btoa(
    JSON.stringify({
      alg: "HS256",
      typ: "JWT",
    })
  );

  const payload = btoa(
    JSON.stringify({
      sub: "1234567890",
      name: "John Doe",
      iat: Date.now() / 1000,
    })
  );

  const signature = generateHS256(`${header}${payload}`);

  const jwt = `${header}.${payload}.${signature}`;
}

function isJWTValid(jwt) {
  const [header, payload, signature] = jwt.split(".");
  return generateHS256(`${header}${payload}`) === signature;
}

function decode(jwt) {
  const [header, payload, signature] = jwt.split(".");
  return JSON.parse(atob(payload));
}

6.4 OAuth 2

• A standard for access delegation
• User registers at an external Identity Provider (IdP), such as Google, Facebook, Amazon, ...
• Authentication is done via the external IdP

An Introduction to OAuth 2

6.5 Auth frameworks

• It's a good idea to use a well established authentication framework
• Passport (JavaScript)
• Authlib (Python)
• Devise (Ruby)
• Goth (Go)

6.6 Input validation

• User input must always be treated as insecure
• All data (URL and HTTP-Body) must be validated before usage (e.g. database access)
• Some protocols already include validation (grpc, ProtoBuf) but HTTP and Websockets don't
• It's a good idea to use a well established library (security and performance)

Custom implementation

in general a bad idea (Do you know all edge cases and language quirks? What about complex validations?)

const isString = (value) =>
  typeof value === "string" || value instanceof String;

const isEmail = (value) =>
  isString(value) && value.match(/^\w+@[a-zA-Z_]+?\.[a-zA-Z]{2,3}$/);

Example libraries for JS/TS

• ajv (JSON schema validation)
• joi (Schema through method chaining)
• class-validator (Typescript classes with decorators)

6.7 Output serialization

• Often the API response varies from the database schema
• Data must be sanitized (e.g. HTML characters)
• Data must be transformed (e.g. remove or rename keys, stringify values, ...)

MongoDB Document

{
  _id: ObjectId("5e1a0651741b255ddda996c4"),
  __version: 2,
  name: 'Jane Doe',
  email: 'jane.doe@example.com',
  hashedPassword: `$2y$10$W/Lrp/XbIYhWbLPe3KdpVePQtwQnWbTm7UxNkgPyyKL7u8wOOv.u2`,
  projects: [Types.ObjectId("6256b6588803043ce532babb"), Types.ObjectId("6256b66841160b21af5d96d0")]
}

Example usage

const { Schema, Types, model } = require("mongoose");

// Mongoose schema
const userSchema = new Schema({
  name: String,
  email: {
    type: String,
    lowercase: true,
    index: true,
  },
  hashedPassword: {
    type: String,
    select: 0, // Exclude password by default
  },
  projects: {
    type: Types.ObjectId,
    ref: "Project",
  },
});
const userModel = model("User", userSchema);

// Mongoose query response
const doc = await userModel({ email: "jane.doe@example.com" });

// Response object
const serializedDoc = {
  id: doc._id.toString(), // Rename _id, stringify ObjectID
  // Remove __version
  name: doc.name,
  email: doc.email,
  projectsCount: projects.length, // Reduce array to integer value
};

Example libraries for JS/TS

• class-transformer

6.8 Error handling

• Catch all errors (you're application should never crash)
• Log all errors
• Don't expose sensitive data to the client

One solution: Custom errors

class ValidationError extends Error {
  public details;

  constructor(message, details) {
    super(message);
    this.details = details;
  }
}

try {
  // Bad
  throw new Error("Validation Error");

  // Better
  throw new ValidationError("Validation Error", details: { /* fields and values that are wrong... */ } );
} catch (err) {
  // Error handler
  if (err instanceof ValidationError) {
    res.status(400).json({ message: err.message, details: err.details });
  } else {
    res.status(500).json({ message: "Internal Server Error" });
    console.error(err);
  }
}

Example libraries for JS/TS

• boom
• nestjs: Exceptions

6.9 Logging

• Use levels like debug or verbose and don't log everything with (info/log)
• Use the ISO 8601 standard YYYY-MM-DDThh:mm:ss.SSSTZD
• Use an application wide logger (standard lib or something custom) that extends all logging metadata
• Log at least all errors
• More data is better than less (except you get performance problems)

Example implementation

This is just an example. Look for existing libraries that fit your scope.

const Config = require('./config');

class Logger {
  private context = 'Unknown';

  constructor(context) {
    this.context = context;
  }

  private console(level, message, ...details) {
    const line = `${new Date().toISOString()} [${level}] ${message} - (${this.context})`;
    switch(level) {
      case 'debug':
        console.debug(line, ...details);
        break;
      case 'info':
        console.info(line, ...details);
        break;
      case 'warn':
        console.warn(line, ...details);
        break;
      default:
        console.error(line, ...details);
        break;
    }
  }

  private json(level, message, ...details) {
    console.log(JSON.stringify({
      level,
      timestamp: new Date().toISOString(),
      message,
      details,
      meta: {
        service: Config.ServiceName,
        version: Config.Version,
        context: this.context,
      }
    }));
  }

  private print(level, message, ...details) {
    if (Config.Environment === 'production') {
      this.json(level, message, ...details);
    } else {
      this.console(level, message, ...details);
    }
  }

  debug(message, ...details) {
    if (Config.Environment === 'production') {
      return;
    }
    this.print('debug', message, details);
  }

  info(message, ...details) {
    this.print('info', message, details);
  }

  warn(message, ...details) {
    this.print('warn', message, details);
  }

  error(message, ...details) {
    this.print('error', message, details);
  }
}

Usage

server.js

const { Logger } = require("./utils/logger");
const logger = new Logger("Application");

logger.info("Application started");
// 1970-01-01T00:00:00.000Z [info] Application started - (Application)

try {
  throw new Error("Something went wrong");
} catch (err) {
  logger.error(err.message, err);
  /**
    1970-01-01T00:00:00.000Z [info] Something went wrong - (Application)
      Something went wrong
        at Object.<anonymous> (.../server.js:1:15)
        at Module._compile (node:internal/modules/cjs/loader:1101:14)
        at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
        at Module.load (node:internal/modules/cjs/loader:981:32)
        at Function.Module._load (node:internal/modules/cjs/loader:822:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
        at node:internal/main/run_main_module:17:47
   */
}

user.controller.js

const { Logger } = require("./utils/logger");
const logger = new Logger("UserController");

logger.debug("Something else");
// 1970-01-01T00:00:00.000Z [debug] Something else - (UserController)

6.10 Security example (expressjs)

In a real application functionality should be separated into different files (e.g. controllers, services, serialization, ...)

const express = require("express");
const cors = require("cors");
const joi = require("joi");
const logger = require("./utils/logger");
const db = require("./utils/db");
const { verify } = require("jsonwebtokn");
const {
  NotFoundError,
  AuthenticationError,
  ForbiddenError,
  InternalServerError,
  GenericError,
} = require("./utils/errors");

const PORT = parseInt(process.env.PORT);
const JWT_SECRET = process.env.JWT_SECRET;

const app = express();

// Parse request body data
app.use(express.json());

// CORS middleware
app.use(cors({ origin: ["localhost:3000"] }));

app.post("/login", (req, res, next) => {
  // login logic
});

// Secure all other routes with a middleware that stores jwt payload into req.user
app.use((req, res, next) => {
  try {
    req.user = verify(req.headers.authorization.replace("Bearer "), JWT_SECRET);
  } catch (err) {
    next(new AuthenticationError("Missing or invalid jwt"));
  }
});

app.post("/products", async (req, res, next) => {
  try {
    // Authorize
    if (req.user.role !== "admin") {
      throw new ForbiddenError("Must be admin to create products");
    }

    // Input validation
    const schema = joi.object({
      name: joi.string().required(),
      price: joi.number().optional().default(0.99),
    });
    const dto = await schema.validateAsync(req.body);

    // Database access
    const product = await db.products.create(dto);

    // Serialize response
    const serializedProduct = {
      id: product._id.toString(),
      createdAt: product.createdAt,
      updatedAt: product.updatedAt,
      name: product.name,
      price: product.price,
    };

    // Response
    res.status(200).json(serializedProduct);
  } catch (err) {
    next(err);
  }
});

// Handle not found routes
app.use((req, res, next) => {
  next(new NotFoundError("Not found", { path: req.path, method: req.method }));
});

// Handle errors
app.use((err, req, res, next) => {
  if (err instanceof GenericError) {
    res.status(err.status).json({ message: err.message });
  } else if (joi.isError(err)) {
    res.status(400).json({ message: err.message, details: err.details });
  } else {
    res.status(500).json({ message: "Internal server error" });
  }
  logger.error(err);
});

// Start server
app.listen(PORT);
logger.info(`Listening on port: ${PORT}`);

7 Tooling

Deployment, testing and operation of applications

7.1 Testing

Disclaimer: Definitions about testing are varying (e.g. what exactly is integration and what is end-to-end...)

General rules

• Tests should be independent to avoid false positives or false negatives (e.g. a create and delete test)
• Unit-tests should mock external function calls (only test logic inside the tested function)
• Side effects should be checked with spys (verify that a function is called)

Backend Testing

• Mocking databases is difficult and might hide errors in your application code
• Running databases in a separate container is also difficult (e.g. for CI jobs)
• Solution: Switch the ORM/ODM adapter for testing to an in-memory or file-based database (e.g. sqlite, MongoMemoryServer)

Backend Unit-Tests

Test small isolated parts of code (dependencies should be mocked and asserted with spys)

• What to test: all parts of code (mostly CRUD service methods), cover all code-branches
• Assertions: return data, errors
• Examples:
  • "ProductService.create() should verify" -> Assert: return { id: 1337, ... }
  • "ProductService.create() with duplicate title should fail" -> Assert: Error("duplicate_title")
  • "ProductService.create() with invalid category id should fail" -> Assert: Error("invalid_category_id")

Backend E2E-Tests

Test a complete request-response cycle with http-requests

• What to test: route exists, route specific middlewares are enabled (e.g. auth guards, input validation, output serialization)
• Assertions: status codes (success and error), response body
• Examples:
  • "POST /products should verify" -> Assert: 201 Created
  • "POST /products without auth should fail" -> Assert: 403 Forbidden
  • "POST /products without invalid body should fail" -> Assert: 400 Bad Request

Frontend Testing

• Mock all API calls (e.g. with a global interceptor)

Frontend Unit-Tests

Test small isolated parts of code (dependencies should be mocked and asserted with spys)

• What to test: all parts of code (mostly business logic), cover all code-branches
• Assertions: return data, errors
• Examples:
  • "StorageService.read() should verify" -> Assert: value is returned correctly from localStorage
  • "StorageService.read() with undefined key should throw" -> Assert: Error("undefined_key")

Frontend Component-Tests

Test the functionality of a component

• What to test: props and state (js) are correctly reflected in the UI (html, css)
• Assertions: html content, classes, attributes, styles
• Examples:
  • "Prop disabled should disable control" -> Assert: input field has attribute "disabled"
  • "Text input value is not a valid email pattern" -> Assert: input field has class "invalid"

Frontend E2E-Tests

Test a use-case of the application by simulating a user flow

• What to test: Application works as expected
• Assertions: no crashes, DOM, URL
• Examples:
  • "Login and logout should verify" -> Start App -> fill out login form -> Submit Form -> should route to Home-Screen -> Press Logout

7.2 Test-driven Development (TDD)

• Writing tests first let's you think about how you want to use your API
• Slower at first, but impossible to work without in bigger projects
• Forces you to structure your code in a testable way

Development Loop

1. Implement a test that calls new functionality
2. Run tests and make sure they fail
3. Implement functionality
4. Run tests and make sure they succeed
5. Go back 1.

Example

Iteration 1 (Write test)

// app.spec.js
test("/products should return status code 200", async () => {
  const res = await fetch("/products");
  expect(res.statusCode).toBe(200);
});

// app.js
const express = require("express");
const app = express();
express.listen(3000);

/** run tests
 * products should return status code 200 ❌
 */

Iteration 2 (Write programm) -> test succeeds

// app.spec.js
test("/products should return status code 200", async () => {
  const res = await fetch("/products");
  expect(res.statusCode).toBe(200);
});

// app.js
const express = require("express");
const app = express();

app.get("/products", (req, res) => {
  res.status(200).send();
});

express.listen(3000);

/** run tests
 * products should return status code 200 ✅
 */

Iteration 3 (Write test) -> test fails

// app.spec.js
test("/products should return status code 200", async () => {
  const res = await fetch("/products");
  expect(res.statusCode).toBe(200);
});

test("/products should return an array", async () => {
  const res = await fetch("/products");
  expect(Array.isArray(res.data)).toBe(true);
});

// app.js
const express = require("express");
const app = express();

app.get("/products", (req, res) => {
  res.status(200).send();
});

express.listen(3000);

/** run tests
 * products should return status code 200 ✅
 * products should return an array ❌
 */

Iteration 4 (Write programm) -> test succeeds

// app.spec.js
test("/products should return status code 200", async () => {
  const res = await fetch("/products");
  expect(res.statusCode).toBe(200);
});

test("/products should return an array", async () => {
  const res = await fetch("/products");
  expect(Array.isArray(res.data)).toBe(true);
});

// app.js
const express = require("express");
const app = express();

app.get("/products", (req, res) => {
  res.status(200).json([]);
});

express.listen(3000);

/** run tests
 * products should return status code 200 ✅
 * products should return an array ✅
 */

7.3 Containers

• A container feels like a super-lightweight virtual machine
• Lightweight: A laptop can run multiple thousands easily
• Multiple containers can communicate with each other
• Containers can be deployed straight into production (Google Cloud, AWS, Azure, Kubernetes, ...)
• Analog to version control for source code, but for your application runtime

7.4 Docker

• docker is the most promiment implementation
• Docker: Containers vs. Virtual Machines

Image

• Contains all data to instantiate a container
• Think of it as a class in Object-Oriented Programming
• An image is build from a specification, e.g. a Dockerfile
• Images have names
• Some of them are prebuild and available for the public e.g. mongo, mysql, node, node-alpine, ...
• Some of them are private, e.g. the ones you build locally

Registry

• Contains public or private images
• Images can be pulled from and pushed to registries
• Most famous is hub.docker.com

Container

• An instance of an Image
• Has a random or given name
• Can be in different states, e.g. running, stopped, failed, ...

Network

• A private network shared by some containers
• Not accessible from the outside, except ports are explicitly mapped to the outside world
• Has a name

Volume

• A folder mounted to a running container
• Without volumes, containers are unable to persist data
• Has a name

Stack

• A composition of multiple containers, networks and volumes
• Specification of a complete application stack
• Think of a receipt: We need one backend container, two database containers, a frontend container, ...
• Specified through docker-compose.yml

7.5 Docker examples

Some examples to show what you can do with docker

Most useful command

docker ps # list all running containers

Run a bash in a container

docker run -ti alpine sh

Mount a volume to a container

mkdir data
echo "hello world" > data/hello
docker run -ti -v $PWD/data:/data alpine sh

Run node REPL in a container (no local install)

docker run -ti node:16-alpine

Run a redis database in a container, access it from another

docker network create mwa
docker run --name redis --network mwa -d redis:4-alpine
docker run -ti --network mwa redis:4-alpine sh

Serve some static content through a container

mkdir public
echo "hello world" > public/index.html
docker run -v $PWD/public:/usr/share/nginx/html -p 8080:80 -d nginx

7.6 Docker Compose

• Compose allows to run multiple containers as defined by a single configurationfile: docker-compose.yml
• The complete stack of containers can then be started using docker compose up

Example

• Starts up backend and frontend service, where each has its individual folder and Dockerfile
• Mapping volumes during development allows to update files in each service without rebuilding it

version: "3.6"
services:
  backend:
    build: ./backend
    ports:
      - "8081:8081"
    volumes:
      - "./backend:/usr/src/app"
  frontend:
    build: ./frontend
    ports:
      - "8080:80"
    volumes:
      - "./frontend/public:/usr/share/nginx/html"

Startup sequence

• Services can express their dependencies using depends_on to influence the startup order
• However, this does NOT mean that the dependet-on service, e.g. a database is ready
• Control startup and shutdown order in Compose
• To ensure a flawless startup, docker compose offers a healthcheck configuration that polls the service to expose a health state

version: "3.8"
services:
  backend:
    build: ./backend
    ports:
      - "3000:3000"
    healthcheck:
      test: curl --fail http://localhost:3000/healthcheck || exit 1
      interval: 5s
      retries: 10
      timeout: 3s
  frontend:
    build: ./frontend
    ports:
      - "8080:4200"
    depends_on:
      backend:
        condition: service_healthy

7.7 Gitlab workflow

issues, branching, merge requests and code-review

Gitlab provides a well established workflow to combine organization (issues, code review) and coding (git branching, merge requests)

Videos can't be printed.

Description: Video showing how to create issues and resolve merge requests

Link: assets/gitlab-merge-requests.mp4

1. Create an Issue
2. As soon as you want to work on the issue, use "Create merge request and branch" through the Gitlab UI
3. Run git pull on your local machine to receive the new branch
4. Checkout the new branch "x-escaped-title-of-the-issue" and start coding
5. Commit and push until everything is implemented
6. Open the Merge-Request in Gitlab UI, assign a reviewer (if not done yet) and mark Merge Request as ready
7. Reviewer should overview all changes and trigger the merge through Gitlab UI (delete source branch and squash commits is practical in most cases)
8. Closing the merge request will automatically close the issue

Go to step 1 or 2 and fix the next issue

Merge-Conflicts?

Gitlab will check on every code push, if the Merge-Request can be merged. If there are merge conflicts, you can either:

• Fix them through Gitlab UI (if it's an easy conflict)
• Merge the newest main/master branch locally in your issue-branch and fix it e.g. with VSCode

7.8 Gitlab CI

Configuration

Gitlab-CI has to be enabled first:

• Activate pipelines: Settings > General > Pipelines > Enable
• Add runners: Settings > CI/CD > Runners > Enable Shared Runners

Example .gitlab-ci.yml

stages:
  - setup
  - test

cache-node-modules:
  stage: setup
  image: node:alpine
  script:
    - npm ci
  artifacts:
    name: node_modules
    paths:
      - node_modules
    expire_in: 1h

unit-tests:
  stage: test
  image: node:alpine
  script:
    - npm run test

e2e-tests:
  stage: test
  image: node:alpine
  script:
    - npm run test:e2e

8 Questions

Are there any further questions?