motcha reblogged at 7 years ago
黒糖の欠片 〜それは名前を盗まれた街の物語〜 BrownSugar@mstdn.maud.io
再試行がtimeoutで溜まり続ける
フォローしようとすると503が出るユーザーが居る
取得できるユーザーは普通に取得できるけど取得できないユーザーは確定でtimeoutする
外部からもフォローに失敗することがあるらしい
motcha mot@mastodon.motcha.tech
Redisは差分が出ちゃったから、なんとか元のデータとがっちゃんこしてなるべく復旧したいけどマージというかなんちゅーかをするメカニズムが効率的に作れてないから時間がかかりそうで…ほうほう…
motcha reblogged at 7 years ago
Eugen Gargron@mastodon.social
What I still need to do is reconfigure the old, now stale postgres master to be a failover replica. But I might have to go a step further and switch them around again after that because I am seeing non-zero iowait on the current master which makes me suspect the SSD is faulty.
motcha reblogged at 7 years ago
Eugen Gargron@mastodon.social
I previously only used feed building in my development environment on super small test data. That procedure is not optimized, it took 10 hours to complete.
motcha mot@mastodon.motcha.tech
ほー。電源障害でぽすぐれとRedisがダウン、レプリカにフェールオーバーしてそっちがマスタに昇格したからDBは問題ないけど、Redisはレプリケーションしてなかったからゼロから再生成されて…元のサーバが復帰したからRedisを元に戻そうと思ったが…？ってとこね
motcha reblogged at 7 years ago
Eugen Gargron@mastodon.social
Okay, so when I left, Hetzner had a power failure and the postgres and redis master server went down. A failover replica was available and was promoted to new master. Redis was not replicated so was started from scratch and home feeds were regenerated.
When the original server went back online, I had to switch the redis server back over, but now had divergent data. I decided to opt for the older, more full data, and just regenerate up-to-date home feeds on top of it.