Chatsubo Wintermute • 1y ago • 100%

[Completed] Updating to 0.18.4

Lemmy 0.18.4 has been released. You can see the official release announcement [here](https://lemmy.villa-straylight.social/post/257480?scrollToComments=true). This one's a small one so should go smoothly. As usual, we'll update at 04:00 UTC tonight. Edit: Completed successfully as expected. Unfortunately we did have a significant downtime just before the update due to the serve running out of memory. This was partially an odd coincidence and partly my fault. I had noticed that we were getting very near being out of memory, but expected the server would hold out until we rebooted during the maintenance window tonight. Unfortunately that didn't happen. Lemmy does gradually eat memory over time and requires periodic restarts, but the frequency seems rather variable. I never had restart the server for the entire time we were running on 0.18.2, while 0.18.3 needed a restart after just a handful of days of operation. Most significantly, this incident revealed that some of my monitoring tools aren't reporting correctly. I'll be working on improving that in the future. I may also institute a periodic server restart as well. As long as I only have to restart Lemmy and not the entire server, this should be quick enough that no one will really notice.

Chatsubo Wintermute • 1y ago • 100%

[Completed] Updating to 0.18.3 2023-07-31

Lemmy 0.18.3 has been officially released. You can read the official release notes [here](https://github.com/LemmyNet/lemmy/blob/main/RELEASES.md). The eagle-eyed among you may notice that we are upgrading from 0.18.1 to 0.18.3 and wonder what happened to 0.18.2, and I share you confusion. It never got an official GitHub release so I never upgraded to it. I plan to update tonight at ~~2023-07-29~~ ~~2023-07-30~~ 2023-07-31 04:00 UTC. Expected downtime is slightly longer than usual due to database updates, but should still be less than 10 minutes. As always, downtime may extend to up to an hour if any thing goes wrong. Edit: There was a glitch with the update and I had to restore from a backup, which took most of the planned hour. I did successfully test the update on a clone of the production VM beforehand so I'm hopeful that simply retrying the upgrade will succeed, but to keep things within my one hour maintenance window I'll hold off on trying again until the same time tomorrow. Edit 2: Sorry folks, same result attempting the update tonight. I did at least get a much better capture of the logs as I was prepared this time. I'm indefinitely postponing the update while I investigate and come up with a solution. If I can repeat the update successfully again on a test server I have some ideas on how to use that as a migration path. Will also be reaching out to other instance admins on Matrix to see if this is a know issue. Will keep you all updated with a new post when I have something new to share. Edit 3: I had an epiphany last night as I was going to bed. I tested it this morning and my guess was correct. Basically I'm just an impatient idiot. This update requires a significant database migration, during which time the lemmy backend doesn't respond to requests. The frontend is up and running though and is able to display en error message. Being cautious about downtime, I saw the error message and defaulted to getting things to a known-good state withing the one hour maintenance window I had published. I suspect I was multi-tasking a bit while testing the upgrade so I just never got around to loading the test instance until after the database migration had completed. Will proceed with the update tonight with an improved understanding of what to expect. Edit 4: Update completed. Everything looks good.

Chatsubo Wintermute • 1y ago • 100%

[Completed] Short downtime at 19:00 UTC today

We are slowly accumulating both image and database data and our current disk will likely be full within the next several days. Fortunately I can increase the size of the disk by another 30GB without incurring any additional cost, which should be enough to *at least* another month of data. Unfortunately Linode doesn't let me increase the size while the server is online so there will be a brief downtime while I shut down the server and reallocate the disk. Edit: Completed with just over 1.5 minutes of downtime.

Chatsubo Wintermute • 1y ago • 100%

[Planned Downtime] Updating to 0.18.1 tonight

Lemmy 0.18.1 has been officially released. You can read the official release notes [here](https://lemmy.villa-straylight.social/post/114675). I plan to update tonight at 2023-07-07 04:00 UTC. Expected downtime is just a couple of minutes, but may extend to up to an hour if any thing goes wrong. Edit: After a slight delay unrelated to the server, the update was uneventful and looks like it was successful.

Chatsubo Wintermute • 1y ago • 100%

[COMPLETED] Updating to 0.18.0

Hi Everyone, This is just a quick FYI that I will be upgrading the instance to the Lemmy 0.18.0 release today at 21:00 (9PM) Pacific time, AKA 2023-06-23 04:00 UTC. Expected downtime is just a few minutes, but may extend up to an hour if anything does not go to plan. You can read the full release announcement [here](https://lemmy.villa-straylight.social/post/62461). Edit: As of 21:03, everything appears to have gone as planned and we are now running 0.18!

Chatsubo Wintermute • 1y ago • 100%

Brief downtime earlier today

I caught this fast enough that I wonder if anyone even noticed, but this is the first unplanned downtime for the instance I'll go ahead and report on it anyway. The instance has been consistently running at approximately 75-80% memory usage since it started. This morning (in my local time zone) the server experienced a memory spike. This spike caused the kernel to start swapping excessively, which in turn spiked CPU usage to 100%. # Timeline of Events * 2023-06-14 17:11 UTC: Memory hit 100%, caused excessive swapping and 100% CPU usage * 2023-06-14 17:14 UTC: Issue was noticed thanks to monitoring tools * 2023-06-14 17:15 UTC: Shut down node, initiated node resize to double RAM and CPU * 2023-06-14 17:17 UTC: Server back up and running # Evaluation I was already a little concerned about running the server so close to the edge of memory. This event proves that it's not tenable. With the new node we now have 4GB of RAM and 2 CPU cores compared to 2GB and 1 core before. We are currently sitting at 36% memory usage and ~6% CPU. In general Lemmy is CPU bound for large user counts, so this server size should support us up to several hundred users. I expect memory will not be a concern for the foreseeable future. As for what caused the event in the first place, I'm not sure. Server logs around the time of the event don't look unusual, but it's not practical to log all inbound activity since every comment and every vote on every instance that we are subscribed to is a separate HTTP request. It's possible that the spike was a result of bursty federation messages from lemmy.ml, which has been struggling under load the last couple of days. I feel pretty confident about the new server hardware for now so I will continue to diligently monitor system performance, but I don't expect this particular issue to recur.

Chatsubo Wintermute • 1y ago • 100%

Say hello

We're getting a slow but steady stream of signups, but most of you are just lurking. Creating at least one post or comment helps boost our active user count, which in turn gets us higher on the join-lemmy.org website so that more people see our little corner of the Fediverse when deciding where to sign up. I don't expect to ever be one of the major servers of the Lemmiverse, but I do think the more people we have here the more fun it will be. To that end, please leave a comment here introducing yourself to the server, especially if you have been lurking up till now. Or, just go out there and comment anywhere on Lemmy (it doesn't have to be on our instance) so that you are counted as active!

Chatsubo Wintermute • 1y ago • 100%

Artists wanted

We're slowly accumulating users here so I figure I will throw this out there. Right now pretty much all of the icons and banners on the instance are AI generated art courtesy of Midjourney. While that has its own appeal given my username, I would be more than happy to replace it with user-submitted improvements (even more AI generated stuff. I'm not all that great at getting what I want out of the AI anyway). In exchange I can offer you a shout out in the community/instance description or something in addition to the karmic benefit of contributing to the instance. :)

Chatsubo Wintermute • 1y ago • 100%

Occasional instance restarts and how to handle them

As you all know, Lemmy is a rapidly evolving platform. That goes for those of us administering instances as well as users. As we grow and learn, it is occasionally necessary to restart the Lemmy instance to tweak settings. Generally speaking, I'm able to set these changes up first, then click the button to restart the server, so the instance is down for less than a minute at a time. I think this is short enough that it's not worth making a post and waiting for an announced time for every restart. When/if I encounter a situation where I need to make more significant changes they will definitely be announced. I would like to hear whether anyone has noticed and/or is bothered by occasional unannounced downtimes. If there's anyone out there that wants me to announce them first, I'm happy to do so when possible. The only cost is that it slows down my ability to iterate towards the ideal server config.

Space Wintermute • 1y ago • 100%

A graphic showing the most powerful rocket launched over time

I'm not above posting reddit links to interesting OC

Space Wintermute • 1y ago • 100%

A graphic showing the most powerful rocket launched over time

I'm not above posting reddit links to interesting OC

Asklemmy Wintermute • 1y ago • 96%

How to get users for a new instance?

I self-hosted an instance a when I got here a few days ago. The server costs me the same whether I use it or not, so I'd like to let as many people sign up as I can. I had a few people sign up already before join-lemmy.org was changed to require 5 monthly active users, but now my instance isn't shown there because it's below the limit. Any idea on how to attract a dozen or so users so that I can show up on the main join-lemmy page again?

Chatsubo Wintermute • 1y ago • 100%

Hosting a Lemmy Instance

A rambling list of notes to myself about hosting a Lemmy instance like this one # Basic Setup Docker guide: https://lemmy.villa-straylight.social/post/1849# However, I like the `docker-compose.yml` in the official docs better as a starting point. The docs pages aren't automatically published when a PR is merged, so refer to the [Github Repo](https://github.com/LemmyNet/lemmy/blob/main/docker/docker-compose.yml) for the latest version. This `docker-compose.yml` is mostly intended for development work, so you will need to replace the following lines: ```yaml build: context: ../ dockerfile: docker/Dockerfile ``` with ``` image: dessalines/lemmy:0.17.4 ``` Likewise, update the `lemmy-ui` image to `dessalines/lemmy-ui:0.17.4`. I use basically the `nginx.conf` added by Just-Insane to the docker install page [here](https://github.com/LemmyNet/lemmy-docs/blob/main/src/en/administration/install_docker.md). # Reverse Proxy Lemmy uses websockets (for now anyway, that's changing soon) and websockets are kind of dumb. If you put this setup behind a reverse proxy, you need to make sure the reverse proxy properly handles web sockets. For `nginx`, this essentially means you need to copy the settings in the docker `nginx.conf` into your fronting reverse proxy. **Example:** ``` server { listen 443 ssl http2; listen [::]:443 ssl http2; server_name lemmy.villa-straylight.social; ssl_certificate /etc/letsencrypt/live/villa-straylight.social/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/villa-straylight.social/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot # frontend general requests location / { proxy_pass "http://localhost:[docker nginx port]"; rewrite ^(.+)/+$ $1 permanent; # Send actual client IP upstream proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } # backend location ~ ^/(api|pictrs|feeds|nodeinfo|.well-known) { proxy_pass "http://localhost:[docker nginx port]"; # proxy common stuff proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; # Send actual client IP upstream proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } ``` # Custom Themes In `docker-compose.yml`, add a new volume to `lemmy-ui`, with `/app/extra_themes` as the path in the image. (Thanks to [@tmpod@lemmy.pt](https://lemmy.pt/u/tmpod) for pointing that out) **Example**: ```yaml lemmy-ui: image: dessalines/lemmy-ui:0.17.3 volumes: - ./volumes/lemmy_themes:/app/extra_themes/ ``` You can then drop .css files in there (e.g. a copy of [one of the default themes](https://github.com/LemmyNet/lemmy-ui/tree/main/src/assets/css/themes)) and they will appear as an option in the user's settings. The name of the theme is the file-name of the .css file.

Cyberpunk Wintermute • 1y ago • 100%

What's your favorite Cyberpunk work

Be it book, film or any other medium, share your favorite works of the genre (and why it's your favorite!)