Apache Airflow 遇到的問題紀錄

1.StatsD 無作用
版本: 2.0.1
Airflow 2.0 does not send metrics to statsD when Scheduler is run with Daemon mode #13741
https://github.com/apache/airflow/issues/13741
原因: daemonContext will close the socket of statsd.
暫時解法: 不使用 airflow scheduler -D, 先改用 airflow scheduler &

Fix statsd metrics not sending when using daemon mode #14454
https://github.com/apache/airflow/pull/14454
版本2.0.2有修復

2.重啟scheduler時發現無法使用daemon啟動
安裝airflow後第一次啟動是可以用daemon的，但後來的重啟都失敗。
版本: 2.0.1

Can not started scheduler when use command "airflow scheduler -D" #10055
https://github.com/apache/airflow/issues/10055

原因:"pooling" behavior .
暫時解法: disabling connection pools
參數設定: sql_alchemy_pool_enabled = False

未修復

Modify: 2021/04/22

Getting a mortgage

A: Hey, Ben, I think I'm finally ready to become a homeowner. B: That's great! Doy you hav your down payment saved up? A: Yup. But I'm not really sure about the mortgage. Should I get a fixed rate or a variable one. B: I'd day get a fixed rate, with an amortization of 15 years. A: That seems like forever. Can I pay it off sooner, like if I get a lummp sum of money? B: You'd better check with the bank. Sometimes they penalize you for that. A: And then how do I know how much house I can afford? I don't wnat to be house poor. B: Generally speaking, your payment shouldn't exceed 25% of your monthly take-home pay. A: Okay, thanks. Hopefully in a few months I'll be moving into my new pad. B: Just make sure to invite me to the housewarming party! homeowner down payment 分期付款的頭期款 mortgage loan fixed rate variable rate amortization 分期償還 (分幾期) fifteen-year mortgage 分15年 lump sum 一次性支付的金額 e.g. Do you wnat us to pay you a little bit every month, or do you want us to pay you in a lump sum, one lump sum? penalize v. 處罰 house poor phr. 房奴 pad n. 住處 bachelor pad 單身漢 stereotypically messys or pool table in the basement or they got a bunch of guy stuff housewarming party 喬遷派對

Git Note

 久久一次換新電腦，要重新設定git環境，紀錄一下日後可以查找

• 設定ssh

  1. 使用ssh-keygen

     D:\>ssh-keygen
     Generating public/private rsa key pair.
     Enter file in which to save the key (C:\Users\itlogs/.ssh/id_rsa):
     Enter passphrase (empty for no passphrase):
     Enter same passphrase again:
     Your identification has been saved in C:\Users\itlogs/.ssh/id_rsa.
     Your public key has been saved in C:\Users\itlogs/.ssh/id_rsa.pub.
     The key fingerprint is:
     SHA256:xxxxx
     The key's randomart image is:
     +---[RSA 2048]----+
     |            .o++o|
     |           .. o+=|
     |      .= .   .   |
     |      o+o        |
     +----[SHA256]-----+
     

  2. 產出了路徑會在 C:\Users\user-name/.ssh/
  3. 設定git使用windows的憑證存放區設定

     git config --global http.sslBackend schannel
     

  4. gitlab設定: user > setting > ssh keys 加入public key (id_rsa.pub)
  5. 取一個遠端數據庫的名稱

     git remote add <name> <url>
     git remote add itlogs https://xxx@gitlab.com/xxx/test.git
     

  6. 推上去

     git push <repository> <refspec>
     git push itlogs master
     

  7. 拉下來

     git pull <repository> <refspec>
     git pull itlogs master
     

  8. clone

     git clone <repository> <directory>
     git clone https://xxx@gitlab.com/xxx/test.git test
     

參考資料:
https://stackoverflow.com/questions/16668508/how-do-i-configure-git-to-trust-certificates-from-the-windows-certificate-store

Docker data path

問題: Docker預設的data path在/vat/lib/docker, 這會隨著container越來越多而佔去許多容量，所以修改路徑。

解決方法: 在/etc/docker底下新增檔案daemon.json寫入

{
  "live-restore": true, #可以stop docker daemon不影響container
  "data-root": "/project/docker",
  "debug": true, #debug log
  "insecure-registries": [],#配置docker的私庫地址
  "registry-mirrors": [
       "https://d8b3zdiw.mirror.aliyuncs.com"
    ],
  "insecure-registries": [
       "https://ower.site.com"
    ]，
}

重啟後生效:
systemctl daemon-reload
systemctl restart docker.service

參考文件: https://docs.docker.com/engine/reference/commandline/dockerd

CentOS 7 journal log

問題: 在查找硬碟空間使用率時，看到/run/log/journal的使用空間4GB，因此來研究

過程:

/run/log/journal 是用來存放執行systemd時的log,

清除時可以清除archice的log, 避免連system.journal也清除，

清除後重啟 systemctl restart systemd-journald.service

清除指令:

．清除超過的時間: journalctl --vacuum-time=30d

．清除超過的空間: journalctl --vacuum-size=3G

設定檔路徑: 

/etc/systemd/journald.conf

其他指令:

journalctl [OPTIONS...] [MATCHES...]

Query the journal.

Flags:

     --system              Show the system journal

     --user                Show the user journal for the current user

  -M --machine=CONTAINER   Operate on local container

  -S --since=DATE          Show entries not older than the specified date

  -U --until=DATE          Show entries not newer than the specified date

  -c --cursor=CURSOR       Show entries starting at the specified cursor

     --after-cursor=CURSOR Show entries after the specified cursor

     --show-cursor         Print the cursor after all the entries

  -b --boot[=ID]           Show current boot or the specified boot

     --list-boots          Show terse information about recorded boots

  -k --dmesg               Show kernel message log from the current boot

  -u --unit=UNIT           Show logs from the specified unit

  -t --identifier=STRING   Show entries with the specified syslog identifier

  -p --priority=RANGE      Show entries with the specified priority

  -e --pager-end           Immediately jump to the end in the pager

  -f --follow              Follow the journal

  -n --lines[=INTEGER]     Number of journal entries to show

     --no-tail             Show all lines, even in follow mode

  -r --reverse             Show the newest entries first

  -o --output=STRING       Change journal output mode (short, short-iso,

                                   short-precise, short-monotonic, verbose,

                                   export, json, json-pretty, json-sse, cat)

     --utc                 Express time in Coordinated Universal Time (UTC)

  -x --catalog             Add message explanations where available

     --no-full             Ellipsize fields

  -a --all                 Show all fields, including long and unprintable

  -q --quiet               Do not show privilege warning

     --no-pager            Do not pipe output into a pager

  -m --merge               Show entries from all available journals

  -D --directory=PATH      Show journal files from directory

     --file=PATH           Show journal file

     --root=ROOT           Operate on catalog files underneath the root ROOT

     --interval=TIME       Time interval for changing the FSS sealing key

     --verify-key=KEY      Specify FSS verification key

     --force               Override of the FSS key pair with --setup-keys

Commands:

  -h --help                Show this help text

     --version             Show package version

  -F --field=FIELD         List all values that a specified field takes

     --new-id128           Generate a new 128-bit ID

     --disk-usage          Show total disk usage of all journal files

     --vacuum-size=BYTES   Reduce disk usage below specified size

     --vacuum-time=TIME    Remove journal files older than specified date

     --flush               Flush all journal data from /run into /var

     --header              Show journal header information

     --list-catalog        Show all message IDs in the catalog

     --dump-catalog        Show entries in the message catalog

     --update-catalog      Update the message catalog database

     --setup-keys          Generate a new FSS key pair

     --verify              Verify journal file consistency

Rancher 無法移動 Namespace 問題

問題:

user不知道做甚麼操作，在project內的namespace被移出了，在他的權限範圍內看不到，所以也無法搬回去。

過程:

登入Rancher後，發現該namespace N1被放置在All, 嘗試move namespace後,出現以下的error

"can't move namespace. Project P1 has resource quota set"

查詢官網文件

https://rancher.com/docs/rancher/v2.x/en/project-admin/namespaces/#moving-namespaces-to-another-project

Note: 說明

．You cannot move a namespace into a project that already has a resource quota configured.

意思是說你不能移動一個namespace, 到已經有設定namespace resource quotes的project下。

解法:

1. 先把該project的namespace resource quotes都不設，清空。

2. 移動該namespace, 就可以搬移了。

3. 再把namespace resource quotes設定回去。

CentOS 憑證路徑

 安裝ca-certificates package:
yum install ca-certificates

啟用動態設定:

update-ca-trust force-enable

加入新的憑證:

cp abc.crt /etc/pki/ca-trust/source/anchors/

設定憑證生效:

update-ca-trust extract

參考:

https://manuals.gfi.com/en/kerio/connect/content/server-configuration/ssl-certificates/adding-trusted-root-certificates-to-the-server-1605.html

CURL 基本指令

 ．查看詳細資訊

curl http://abc.com --verbose

* About to connect() to abc.com port 8080 (#0)
*   Trying 10.0.0.1...
* Connected to abc.com (10.0.0.1) port 8080 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt  <== 吃憑證的路徑
  CApath: none

．帶入憑證

curl --cacert RootCA.crt --cert ClientCA.pem:passwd http://abc.com

．不驗證憑證

curl -k

curl --insecure

．取得網站憑證資訊

echo QUIT | openssl s_client -connect abc.com:8080 -servername abc.com -showcerts

openssl s_client -connect abc.com:8080 -servername abc.com -showcerts </dev/null 2>/dev/null | sed -n '/^-----BEGIN CERT/,/^-----END CERT/p'

openssl s_client -connect abc.com:8080 -servername abc.com -showcerts </dev/null 2>/dev/null | awk '/^-----BEGIN CERT/,/^-----END CERT/'

openssl s_client -connect abc.com:8080 -servername abc.com -showcerts </dev/null 2>/dev/null | sed -n '/^-----BEGIN CERT/,/^-----END CERT/p' > example.crt

．變成設定檔方式

abc.conf
cacert = "RootCA.crt"
cert = "ClientCA.pem:mypasswd"
url = "https://mail.ichiayi.com/t.txt"

curl --conf testcurl.conf

參考:

https://blog.miniasp.com/post/2020/08/23/curl-and-self-signed-certificated-or-CA-certificate

https://www.ichiayi.com/wiki/tech/curl_ssl

Kubernetes Pod cannot allocate memory

問題:

在檢查pod event時，出現以下的error log:

unable to ensure pod container exists: failed to create container for [kubepods burstable pod52610281-d2fe-438b-bc34-ec6c18621a99] : mkdir /sys/fs/cgroup/memory/kubepods/burstable/pod52610281-d2fe-438b-bc34-ec6c18621a99: cannot allocate memory

處理過程:

第一時間看到錯誤集中出現在某一台node上，先cordon有問題的node, 不要再讓pod被排程進來，pod schedule到其他node上是好的，所以針對該node查找問題。

檢查:

$ cat /sys/fs/cgroup/memory/kubepods/memory.kmem.slabinfo
Input/output error  # 無memory leak

$ cat /sys/fs/cgroup/memory/kubepods/memory.kmem.slabinfo

slabinfo - version: 2.1
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
# 有memory leak

解決方法:

1. 這個問題是cgroup bug(memory leak)最快的方式就是，Drain後重啟該node,不過一陣子後還是會發生。

2. 更新kernel到5.14.7-1.el7.elrepo.x86_64 後解決。

其他方式可參考: https://zhuanlan.zhihu.com/p/343031257

Redis Note

．查看Redis 資訊

$ redis-cli info
[...]
# Replication
role:slave
master_host:192.168.50.11
master_port:6379
master_link_status:down

．修改Redis master/slave mode

$ redis-cli
redis 127.0.0.1:6379> slaveof no one

redis 127.0.0.1:6379> info

[...]

# Replication

role:master

connected_slaves:0

https://www.claudiokuenzler.com/blog/470/how-to-do-redis-master-slave-cluster-switch-failover

AWS Note

 EC2:

．Change hostname

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-hostname.html

．Install docker

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html

Linux 修改時區, JVM 異常

．java process 認的時區設定位置在/etc/localtime,不是/bin/date

．因為改完時區後，檢查的方式是下 date -R,這個方式是看/bin/date的時間，所以看是正常的，

但java process是認/etc/localtime, 而這個檔案上次修改被設定softlink指到/usr/share/zoneinfo/Europe/Moscow, 所以在怎麼重啟java 還是沒有效，將softlink移除，重啟java後恢復正常。

阿里雲不同帳號-同地區-內網互通設定

集團為了拆帳，不同分公司分了不同帳號，有需求要互通，設定如下:
A: aaa@aaa.com  香港
B: bbb@bbb.com 香港

方法一: 透過console設定
A帳號的部分設定如下:

1. 授權對象填入B帳號的SG或是IP.

2. 帳號ID填入B帳號的ID(非登入帳號)，在帳號管理裡會有帳號ID。

B帳號則填入A帳號的資訊。

方法二:使用API的方式

帳號	帳號ID	安全性群組	安全性群組ID
帳號A	a@aliyun.com	sg1	sg-bp1azkttqpldxgtedXXX
帳號B	b@aliyun.com	sg2	sg-bp15ed6xe1yxeycg7XXX

1. 在sg1安全性群組中添加允許存取sg2安全性群組入方向的規則。
aliyun ecs AuthorizeSecurityGroup --SecurityGroupId sg-bp1azkttqpldxgtedXXX --RegionId cn-qingdao --IpProtocol all  --PortRange=-1/-1. --SourceGroupId sg-bp15ed6xe1yxeycg7XXX --SourceGroupOwnerAccount b@aliyun.com --NicType intranet

2. 在sg2安全性群組中添加允許存取sg1安全性群組入方向的規則。
aliyun ecs AuthorizeSecurityGroup --SecurityGroupId sg-bp15ed6xe1yxeycg7XXX --RegionId cn-qingdao --IpProtocol all  --PortRange=-1/-1. --SourceGroupId sg-bp1azkttqpldxgtedXXX --SourceGroupOwnerAccount a@aliyun.com --NicType intranet

註: 主要關注的參數為SecurityGroupId、SourceGroupId和SourceGroupOwnerAccount。


參考資訊：
https://www.alibabacloud.com/help/tc/doc-detail/40597.htm#section-bpp-qvf-ip5
https://www.alibabacloud.com/help/tc/doc-detail/25385.htm
https://help.aliyun.com/knowledge_detail/38163.html