Server issues today
 
Notifications
Clear all

Server issues today

(@teebling)
Noble Member

Hey everyone,

You'll have noticed a lot of server errors - 500s and 404s today.

This is because I'm hitting memory caps on my shared hosting and their procwatch daemon is literally killing of processes from barrens.chat all the time.

I am currently in the process of moving the site to a managed VPS with more RAM for the moment to give some breathing room whilst I debug where this high memory usage is coming from.


The site may go down completely for a bit and I don't really know when this will happen either - but don't worry - I have everything backed up including the database.

Really sorry it's been a shitty experience for you today, doing my best to bring things back under control.

Your boy Teeb


Update:

We're on the VPS now.

There's hundreds of 35.x.x.x Google Cloud IP addresses hitting the site a lot at the moment which is what may have caused the spike in memory leading to the failures earlier. So not really a RAM issue due to real traffic, more to do with whatever the fuck this bot or person is trying to do to the site. It's not a security concern because I've talked with the sysadmins and they say they'd see that and haven't yet.

If they're still there tomorrow morning I'll consider block IP banning that range (have been advised that if it is a google managed IP range this won't affect anyone legitimate trying to access the site).

Anyway it's 2am here so I have to go to sleep.

Please guys if you carry on experiencing any issues LET ME KNOW HERE so that I can look into it more tomorrow. Any 500 errors 'page not found' or any of that crap from earlier today let me know.


Another update, looks like a DDoS attack. Why someone would want to deny service to this place is beyond me, really.

Going to get some sleep now (I mean it this time lol), hosts are aware. Might have to get cloudflare up tomorrow or something.

Quote
Topic starter Posted : 29/05/2019 10:04 am
(@jon-bloodspray)
Estimable Member

Ha! Was just about to report this.

ReplyQuote
Posted : 29/05/2019 10:05 am
(@escalotes)
Estimable Member

The requested post could not be found.

ReplyQuote
Posted : 29/05/2019 10:59 am
(@neofrag)
Eminent Member

We need a custom 5XX error-page. Something that says/looks like the wow login-queue.

ReplyQuote
Posted : 29/05/2019 11:09 am
(@teebling)
Noble Member

Welp, looks like we're hitting memory limits. I've been expecting this - may have to move to a new server boys.
Thank you for reaching out to us on Twitter, I wanted to follow up with
you here.

I took a look at your server and see what was causing the 500, 404 and
latency issues you are reporting. Looking into your error logs, I'm
showing a bunch of premature end of script headers, specifically for
viewtopic.php.

[Wed May 29 10:23:36 2019] [error] [client 34.90.112.229] Premature end
of script headers: viewtopic.php

After further investigation, your site is hitting procwatch for excessive
memory usage for a total of 14730 times.

Wed May 29 11:04:24 2019 procwatch3 INFO: PID 20079 (php71.cgi)
thobru14:pg9280716 - 44.7MB ram, 3.66 sec cpu: killed for uid ram (total
1148.8 exceeds limit 500.0; score -44.66)

The below information is the number of times the files or directories are
being called/visited the most:

<lots of blablabla>

Since it appears that you have a significant amount of traffic to your
site, I would highly suggest moving your site to a VPS to help with
performance and stability. I would recommend at least a 2GB RAM VPS.

https://www.dreamhost.com/hosting/vps/

ReplyQuote
Topic starter Posted : 29/05/2019 11:12 am
(@escalotes)
Estimable Member

Time to start running ads, teebs!

ReplyQuote
Posted : 29/05/2019 11:16 am
(@teebling)
Noble Member

Time to start running ads, teebs!

Ewww... never.

ReplyQuote
Topic starter Posted : 29/05/2019 11:51 am
(@acrmojt)
Trusted Member

Check out Scaleway. They have the simplicity of DigitalOcean but the prices are much better.

https://www.scaleway.com/en/pricing/

ReplyQuote
Posted : 29/05/2019 2:03 pm
(@teebling)
Noble Member

Alright guys I'm trialling a month of VPS with Dreamhost to see if 1GB RAM is enough. You may see a blackout as the transfer happens (and when this happens I don't know).

If not I'm going to have to do some serious debugging for memory usage.

ReplyQuote
Topic starter Posted : 29/05/2019 3:20 pm
(@teebling)
Noble Member

Update:

We're on the VPS now.

There's hundreds of 35.x.x.x Google Cloud IP addresses hitting the site a lot at the moment which is what may have caused the spike in memory leading to the failures earlier. So not really a RAM issue due to real traffic, more to do with whatever the fuck this bot or person is trying to do to the site. It's not a security concern because I've talked with the sysadmins and they say they'd see that and haven't yet.

If they're still there tomorrow morning I'll consider block IP banning that range (have been advised that if it is a google managed IP range this won't affect anyone legitimate trying to access the site).

Anyway it's 2am here so I have to go to sleep.

Please guys if you carry on experiencing any issues LET ME KNOW HERE so that I can look into it more tomorrow. Any 500 errors 'page not found' or any of that crap from earlier today let me know.

ReplyQuote
Topic starter Posted : 29/05/2019 6:11 pm
(@dolamite)
Estimable Member

Thanks for working your ass off so the rest of us can continue to argue :P ... but seriously, get some rest man. You've earned it.

ReplyQuote
Posted : 29/05/2019 6:14 pm
(@teebling)
Noble Member

Another update, looks like a DDoS attack. Why someone would want to deny service to this place is beyond me, really.

Going to get some sleep now (I mean it this time lol), hosts are aware. Might have to get cloudflare up tomorrow or something.

ReplyQuote
Topic starter Posted : 29/05/2019 7:46 pm
(@acrmojt)
Trusted Member

Another update, looks like a DDoS attack. Why someone would want to deny service to this place is beyond me, really.

Going to get some sleep now (I mean it this time lol), hosts are aware. Might have to get cloudflare up tomorrow or something.

Are you positive it isn't just Googlebot or something else crawling the site?

ReplyQuote
Posted : 29/05/2019 9:24 pm
Azelkith (Knoxtane)
(@knoxtane)
Estimable Member Moderator

teebling Have a donation page with special ranks. like 4 tiers. $5 Tier 1, $10 Tier 2 and so on. I'd gladly donate to help keep this site a float and to get a cool rank/image near my name.

ReplyQuote
Posted : 29/05/2019 9:59 pm
(@teebling)
Noble Member

Another update, looks like a DDoS attack. Why someone would want to deny service to this place is beyond me, really.

Going to get some sleep now (I mean it this time lol), hosts are aware. Might have to get cloudflare up tomorrow or something.

Are you positive it isn't just Googlebot or something else crawling the site?

Yep - Googlebot has a unique IP and even has an account in barrens chat. It crawls conservatively as per robots.txt - this is certainly not google’s web crawler it’s someone abusing their commercial cloud like a botnet.

ReplyQuote
Topic starter Posted : 30/05/2019 2:04 am
(@ironbrutzler)
Reputable Member

@teebling Have a donation page with special ranks. like 4 tiers. $5 Tier 1, $10 Tier 2 and so on. I'd gladly donate to help keep this site a float and to get a cool rank/image near my name.

Yeah if you need Help with the Cost i and i think some others will help.

ReplyQuote
Posted : 30/05/2019 3:08 am
 kip
(@kip)
Eminent Member

Throw us some ads!

ReplyQuote
Posted : 30/05/2019 5:18 am
(@bowner)
Eminent Member

Glad to see its fixed, had the issue other day. Wasn't sure if had to clean cookies/cache cause i remember seeing a thread mention we have to do that for here iirc

ReplyQuote
Posted : 30/05/2019 12:29 pm
 Casp
(@casp)
Eminent Member

Looks like the website had its own type of 'stress test' :confused:

ReplyQuote
Posted : 30/05/2019 1:37 pm