ブログのインフラ刷新; 格安ECSから格安GKEへ

ECS で運用していたこのブログのバックエンドを GKE へ移管しました

GitHub - miyamo2/infra.miyamo.today: Cloud infrastructure for `miyamo.today`.

Cloud infrastructure for `miyamo.today`. Contribute to miyamo2/infra.miyamo.today development by creating an account on GitHub.

github.com

モチベーション

Kubernetes のキャッチアップ

実務で1年ほど EKS を利用しているため、K8s を本格的に身につけたい
よりコスパよく運用したい

元々最小スペックの Fargate Spot を使ったり ALB の代わりに API Gateway × CloudMap を使ったりコストを抑える工夫をしていたもののなんだかんだで月$30程度かかっていた
春の訪れを感じたから

そんな中 GKE を月$8で運用するTipsを見つけて、これを機に移行を決意しました

実際には NodePool を t2d-standard-2 x2 としたため$8を少し超える費用になりますが、それでも ECS on Fargate Spot より安価に抑えられそうです

ChatGPTには不可能だと言われたけど、月8ドルで運用可能な格安KubernetesクラスタをGCPで構築してみた

zenn.dev

激安GKE部分は上記記事を投稿された方の Pulumi 実装をほぼそのまま Terraform に移植しているため、当ブログ独自のモジュール諸々と変数を適宜削って貰えればある程度流用できると思います

GitOps

K8sへの移行をするにあたり今回新たにマニフェスト管理用のリポジトリを作成しました

GitHub - miyamo2/manifest.miyamo.today: manifest files for blogapi.miyamo.today

manifest files for blogapi.miyamo.today. Contribute to miyamo2/manifest.miyamo.today development by creating an account on GitHub.

github.com

CD には ArgoCD を採用し、App of Appsパターンを用いてマイクロサービス群/BFFでデプロイをグルーピングしています

Copy

.
├ apps/                          # 子App
│ ├ federator/federator.yaml
│ └ microservice/
│   ├ article-service.yaml
│   ├ blogging-event-service.yaml
│   └ tag-service.yaml
└ argo-installs/                 # 親App
  ├ federator.yaml
  └ microservice.yaml

argo-installs/microservice.yaml

Copy

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argo-app-microservice
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: '5'
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://github.com/miyamo2/manifest.miyamo.today.git
    targetRevision: main
    path: 'apps/microservice'
  syncPolicy:
    automated: {}

argo-installs/federator.yaml

Copy

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argo-app-federator
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: '10'
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://github.com/miyamo2/manifest.miyamo.today.git
    targetRevision: main
    path: 'apps/federator'
  syncPolicy:
    automated: {}

apps/microservice/article-service.yaml

Copy

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: article-service
  name: article-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: article-service
  template:
    metadata:
      labels:
        app: article-service
    spec:
      containers:
        - image: us-west1-docker.pkg.dev/blogapi-miyamo-today/article-service/article-service:<タグ>
          name: article-service
          ports:
            - containerPort: 3000
          resources:
            limits:
              memory: 128Mi
              cpu: 32m
            requests:
              cpu: 20m
          readinessProbe:
            grpc:
              port: 80
          livenessProbe:
            grpc:
              port: 80
          startupProbe:
            grpc:
              port: 80
            failureThreshold: 30
            periodSeconds: 10
            timeoutSeconds: 1
          envFrom:
            - secretRef:
                name: secret-article-service
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: article-service
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: article-service
  name: article-service
spec:
  ports:
    - port: 80
  selector:
    app: article-service
  type: ClusterIP
# secret example
# ---
# apiVersion: v1
# kind: Secret
# metadata:
#   name: secret-article-service
# data:
#   COCKROACHDB_DSN: foo
#   SERVICE_NAME: bar
#   NEW_RELIC_CONFIG_APP_NAME: baz
#   NEW_RELIC_CONFIG_LICENSE_KEY: qux
#   PORT: foobar

apps/federator/federator.yaml

Copy

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: federator
  name: federator
spec:
  replicas: 2
  selector:
    matchLabels:
      app: federator
  template:
    metadata:
      labels:
        app: federator
    spec:
      containers:
        - image: us-west1-docker.pkg.dev/blogapi-miyamo-today/federator/federator:<タグ>
          name: federator
          ports:
            - containerPort: 80
          resources:
            limits:
              memory: 128Mi
              cpu: 64m
            requests:
              cpu: 20m
          startupProbe:
            httpGet:
              path: /health
              port: 80
            failureThreshold: 30
            periodSeconds: 10
            timeoutSeconds: 1
          envFrom:
            - secretRef:
                name: secret-federator
            - secretRef:
                name: secret-aws-credentials
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: federator
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: federator
  name: federator
spec:
  ports:
    - port: 80
  selector:
    app: federator
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: federator
  annotations:
    argocd.argoproj.io/ignore-healthcheck: "true"  # ヘルスチェックがprogressのまま変わらないので暫定対応
spec:
  ingressClassName: "nginx"
  rules:
    - host: "*.miyamo.today"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: federator
                port:
                  number: 80
# secret example
# ---
# apiVersion: v1
# kind: Secret
# metadata:
#   name: secret-federator
# data:
#   NEW_RELIC_CONFIG_APP_NAME: foo
#   NEW_RELIC_CONFIG_LICENSE: bar
#   ARTICLE_SERVICE_ADDRESS: baz
#   TAG_SERVICE_ADDRESS: qux
#   BLOGGING_EVENT_SERVICE_ADDRESS: foobar
#   PORT: foobaz

マニフェストの更新フロー

イメージタグの更新はバックエンドのモノレポから repository-dispatch を用いて manifest.miyamo.today のワークフローをトリガーし、自動的にマニフェストを更新する仕組みを取っています

バックエンド側ワークフロー

合計4つのサービス（3 マイクロサービス + 1 BFF）で共通のカスタムアクションを使い、Dockerイメージのプッシュ後に manifest.miyamo.today 側を呼び出します

Copy

name: "Deploy"
inputs:
  target:
    description: "target to deploy"
    required: true
  gcp_credentials:
    description: "SSIA"  # ちゃんと書く
    required: true
  gcp_project:
    description: "SSIA"
    required: true
  gcp_region:
    description: "SSIA"
    required: true
  app_id:
    description: "SSIA"
    required: true
  app_private_key:
    description: "SSIA"
    required: true
  commit_sha:
    description: "SSIA"
    required: true
  owner:
    description: "SSIA"
    required: true
runs:
  using: "composite"
  steps:
    - name: Checkout code
      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

    - id: auth
      uses: google-github-actions/auth@71f986410dfbc7added4569d411d040a91dc6935 # v2.1.8
      with:
        credentials_json: ${{ inputs.gcp_credentials }}

    - name: Set up Cloud SDK
      uses: google-github-actions/setup-gcloud@77e7a554d41e2ee56fc945c52dfd3f33d12def9a # v2.1.4

    - name: Get Registry Host
      id: get-registry-host
      shell: bash
      run: |
        echo "host=${{ inputs.gcp_region }}-docker.pkg.dev" >> $GITHUB_OUTPUT

    - name: Configure docker for artifact registry
      env:
        HOST: ${{ steps.get-registry-host.outputs.host }}
      shell: bash
      run: |
        gcloud auth configure-docker ${{ env.HOST }}

    - name: Build & Push Docker Image
      working-directory: ${{ inputs.target }}
      env:
        ENV_NAME: prod
        REGISTRY: ${{ format('{0}/{1}', steps.get-registry-host.outputs.host, inputs.gcp_project) }}
        REPOSITORY: ${{ inputs.target }}
        IMAGE_NAME: ${{ inputs.target }}
        TAG: ${{ inputs.commit_sha }}
      shell: bash
      run: |
        docker build -t ${{ env.REGISTRY }}/${{ env.REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.TAG }} -f ./.build/package/Dockerfile .
        docker push ${{ env.REGISTRY }}/${{ env.REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.TAG }} 

    - name: Generate a token
      id: generate-token
      uses: actions/create-github-app-token@d72941d797fd3113feb6b93fd0dec494b13a2547 # v1.12.0
      with:
        app-id: ${{ inputs.app_id }}
        private-key: ${{ inputs.app_private_key }}
        owner: miyamo2

    - name: Dispatch manifest update
      uses: peter-evans/repository-dispatch@ff45666b9427631e3450c54a1bcbee4d9ff4d7c0 # v3.0.0
      with:
        token: ${{ steps.generate-token.outputs.token }}
        repository: miyamo2/manifest.miyamo.today
        event-type: ${{ inputs.target }}
        client-payload: |-
          {
            "target": "${{ inputs.target }}",
            "tag": "${{ inputs.commit_sha }}"
          }

manifest.miyamo.today側のワークフロー

client-payload で受け取った情報を基に対象サービスのマニフェストに含まれる Deployment のイメージタグを更新しています
yq が煩雑になるのを避けるため、1つのマニフェストファイルに1つのDeployment を定義して管理しています

Copy

name: CD

on:
  repository_dispatch:
    types:
      - article-service
      - tag-service
      - blogging-event-service
      - federator

permissions: write-all

concurrency:
  group: ${{ github.workflow }}

jobs:
  apply:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

      - name: Get Tag
        id: get_tag
        run: |
          TAG=$(echo ${{ github.event.client_payload.tag }})
          echo "tag=$TAG" >> $GITHUB_ENV

      - if: github.event.client_payload.target == 'article-service'
        run: echo "file=apps/microservice/article-service.yaml" >> $GITHUB_ENV

      - if: github.event.client_payload.target == 'tag-service'
        run: echo "file=apps/microservice/tag-service.yaml" >> $GITHUB_ENV

      - if: github.event.client_payload.target == 'blogging-event-service'
        run: echo "file=apps/microservice/blogging-event-service.yaml" >> $GITHUB_ENV

      - if: github.event.client_payload.target == 'federator'
        run: echo "file=apps/federator/federator.yaml" >> $GITHUB_ENV

      - name: Update tag
        env:
          IMAGE: ${{ secrets.GCP_REGION }}-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/${{ github.event.client_payload.target }}/${{ github.event.client_payload.target }}:${{ env.tag }}
        uses: mikefarah/yq@8bf425b4d1344db7cd469a8d10a390876e0c77fd # v4.45.1
        with:
          cmd: yq eval -i '(. | select(.kind == "Deployment") | .spec.template.spec.containers[] | select(.name == "${{ github.event.client_payload.target }}").image) = "${{ env.IMAGE }}"' ${{ env.file }}

      - name: commit
        env:
          TAG: ${{ env.tag }}
        run: |
          git config --local user.email "41898282+github-actions[bot]@users.noreply.github.com"
          git config --local user.name "github-actions[bot]"
          git add ./apps
          commit_message="chore(${{ env.file }}): bump tag to ${{ env.TAG }}"
          git commit -m "$commit_message"
          git push origin HEAD:main