NewsMag 1.7.1 killed our server again !

Hi, well … it looks like there is some fundamental incompatibility with the 1.7x fork of NewsMag and/or Visual Composer – and our system. I was having repeat server hiccups the past 14 days on/off again, which I thought “might” be related to the update we did of mod_pagespeed around the same time as installing NewMag 1.6 on one of our two highest volume sites.

After numerous “turn things off and see what happens” … I ended up switching the high production site back to an icky version of our old Woo Canvas layout without all our customizations (e.g., it won’t keep sidebar on iPad portrait, home page sucks, etc.) — and we rolled back mod_pagespeed one version. Server returned to normal.

Only normal minor php green to yellow doing something like regenerating 10 years of content thumbnails or republishing 15,000 pages of static content (.shtml/SSI) on our #1 site (not running WP, except old version of Newspaper for my personal blog with 10 entries or less!).

Okay — so, today, thinking we’re back in black — I installed NewsMag 1.7.1B, switched on Visual Composer, switched on Newsmag, switched back on our home page from draft, set that as static front page. Within 30 min. server went from typical ‘2″ load to “32” load and starting kicking me CPU warnings.

Time: Sun May 3 18:41:33 2015 -0400
1 Min Load Avg: 22.51
5 Min Load Avg: 10.53
15 Min Load Avg: 6.12
Running/Total Processes: 31/1043

As the above went to “32” while I was refreshing on server console, I had to restart Apache, then “Killed” visual composer, switched theme back to Woo Canvas (waaaah! Canvas….) …. server back down to normal:

CPU usage
Last 1 minute 1.02
Last 5 minutes 1.19
Last 15 minutes 1.06

Not sure yet what to do. NewsMag 1.4 and VC were rock steady, 1.5/1.6 started to see some increase in issues, 1.7 in any flavor kills us entirely and is not usable for our particular system.

My options right now seem to be:
a) try using theme without Visual Composer, try the custom home page blocks method with widgets; see if server will eat that without choking
b) build a new theme ourselves based on bootstrap 3 + siteorigin … (waaaah!)
c) go back to hacking a custom version of Woo Canvas (waaaaah!)
d) go on vacation and let everything just melt and go out of business … (hmmm… tempting …)

So, was supposed to be working on my ecom system on Sunday, but dealt with this for six hours. Too much fun.

Not much of a simple solution on this one, for me, at least.

I suppose I could switch on the 1.71 theme without VC and see what happens…. hmmm…..\

==================
CPU GenuineIntel, Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
Version Parallels Plesk Panel v11.5.30_build115130819.13 os_RedHat el6
OS CentOS 6.5 (Final)
System Uptime: 202 day(s) 00:35
PHP 5.4.22-28.el6.art
MySQL 5.5.33-20.el6.art

Seems like maybe I need PHP 5.5 to use the new version of these things.

• This topic was modified 11 years by simchris.

Chris, I feel your pain… Same thing we were talking about (re server load) since Friday.

We are having the same problem with v1.71… maxing out resources we had to up memory in php.ini to 2048M just to function without to many errors. But not sure we can handle normal traffic of about 40k – 60k uniqe users per day.

The option to go back to old theme is not possible. We had to go head in on this one, with all the changes made to make our content fit NewsMag its just not an option.

Should we roll back to 1.6? We originally launched with 1.7 so I can not say for sure whether 1.6 would be stable.

• This reply was modified 11 years by mrmorrison.

Hi Chris or Radu…

Do you recommend us reverting to v1.6? Our traffic is dismal and I can only attribute it to the server load and assumption that users are affected and going elsewhere.

Not sure how long till a fix is coming; nor whether we can hold out till then.

Thanks,

Any updates on this? Don’t want to deploy with these kinds of issues!

Hi, Radu
got your emails; will address this once I deal with my client stuff today on the “A” side of the company; where this other stuff is the “B” side.

Other folk — 1.6 works “okay” on our “medium traffic” sites, although 1.5 was rock solid. Still, 1.71 (or 1.7) immediately took the CPU into a death spiral. (Well, technically I think it has to go to 170 to die completely, but going from 1.2 to 32 in 10 minutes from activating the theme …. pretty obvious….)

Although I suspect some other things might be “involved” in these issues, including WP 4.2x and Yoast SEO, and mix of plugins. We’re still getting php “hot spots” today, while running 1.6 on 3 other sites. Main one back to Canvas (yuck!).

I’ll work with Radu to play with this as I can likely mess with one of our less important sites to debug.

Thank Chris

I thought it was Yoast and tested extensively with and with out.. Same spikes with WordPress SEO disabled for 2 days over the weekend.

Now I have to find a version of 1.5

Radu or anyone, what functions would be lost going backwards to 1.5? Are there instructions for rolling back versions? Specifically I don’t want to lose pages or settings.

Thanks again.

You can roll back without losing most settings except some widget go wonky, and home page might need to be re-saved, or checked for module/shortcodes not found in an earlier iteration — I think some codes changed between 1.5 and 1.6 if memory serves. Likely might need to refresh/tweak home page (not big deal really), and might need to drag some widgets back from “disabled” to the right places.

I did a backtrack from 1.7 to 1.6 when 1.7 first released, as our server went up to something like 150 or about 100x normal load (!). Simple. Luckily WordPress “transient” settings and dbase allow you to switch between themes and most settings recover — again except sometimes the widgets, and the menus – might need to go back to the “choose locations” and that takes 2 clicks.

I am going to try Radu’s “test shim” with one of our lesser sites running 1.6 — one of the sites I was baby-stepping before tackling the BIG one. Basically the site which dies with 1.7 has about 50x the traffic of the smaller sites, and because it’s been online for a decade it has a LOT of content, images, etc.

One nice thing about the “related posts” plugin I’m using, is that it only checks the past 365 days, not the past 3,500 days — and caches the result — so a tad more efficient on that bit of query.

More later …. can you say “shenanigans” (my word for 2015 …)

O–M–G — it works! (!!)

Radu, your “shim” 1.0 BETA 2, worked wonderfully. Visual Composer not even on the domain (!).

CPU is “normal” SO FAR (I am double checking every 10 minutes; about to do NEW post to see what happens).

Sent you the long winded email (12:34pm PDT 5/6/15) with server info, and such; and the testing steps I did so far today with Newsmag 1.71B, SHIM 1.0 B2, speed booster on, mod_pagespeed OFF.

I am turning sidebars on/off, changing modules on sidebar to “mess” with it. Leaving home page as the “fancy” layout I already had as page from 2.5 weeks back when setup.

Only thing I’m doing slightly different at the moment is I have the built-in “related” posts OFF and using my long-time fallback “contextual related posts” plugin to just stick 3 text-only related item headlines bottom of each story (it checks content not just tags for last 365 days, and caches the output… nice).

Will update by email AND here, as I play with this more.

You have “rocked” my world today 🙂

Chris

============
SHIM active with 1.71B … 20 minutes and counting….

CPU usage
Last 1 minute 0.52
Last 5 minutes 0.55
Last 15 minutes 0.58

That’s a wee bit better than “32.0” methinks 🙂

• This reply was modified 11 years by simchris.

And anecdotal, halfway through regenerating 5,000 thumbnails (which I’d had to regen on Monday when going back to Woo Canvas temporarily) …

CPU
Last 1 minute 1.40
Last 5 minutes 1.28
Last 15 minutes 0.95

again — far cry from “32.0” with 1.71 + VC 4.4.4 this past Sunday!

SHIM O JOY! 🙂

This sounds like GREAT NEWS guys!!

Sorry I was not able to help with testing the shim.. I have been trying to play catchup. Reverting from v1.71 to 1.6 eased much of the stress on our server.

Also, It is way outside my technical expertise to run the tests you needed, although I would be willing to let you Radu (or Chris for that matter) access our back-end (no pun intended).

Otherwise anything I CAN do to help I am happy too.

HI, just to give you an idea ….
our site CaliforniaNewswire.com running the shim “beta 2” is right now trouble free; although I have not yet switched on mod_pagespeed but will be doing that today — it shouldn’t cause any issues.

By comparison, one of our other sites eNewsChannels.com – running 1.6 + VC got a huge influx of traffic last night, and some scraping from various syndicators, and our CPU went to

Time: Wed May 6 23:12:58 2015 -0400
1 Min Load Avg: 53.57
5 Min Load Avg: 14.23
15 Min Load Avg: 5.33
Running/Total Processes: 107/1111

Based on the Apache error log, that was all from that ONE site, being scraped from “54.160.202.74”

9-0 22697 0/51/52008 W 9.85 3 0 0.0 1.12 399.32 54.160.202.74 enewschannels.com HEAD /news/news-made-in-the-usa/ HTTP/1.1

One index page being scraped for all the headlines, used 9.85% of the CPU (!). And, um them be like QUAD XEON processors, too.

So, it’s pretty clear Visual Composer when ‘under load’ really whacks the system!

Chris, Thats awesome.. Even tho I dont have the tech know-how to run the tests Radu wants.. I have the shim and will activate it tonight and see what happens..

Radu,
I sent you copy of the error logs generated automatically (light versions), for the ‘overload’ which happened overnight from the ONE website — which is still running VC + NM 1.6 — just for reference. Anecdotal, and not necessarily useful from the standpoint of all server processes running, but specific to what was causing CPU spikes last night for one hour/period to generate the warning/alert email from server.

Look for email with subject
SHIM BETA 2 PROJECT / blah blah.
Sent that at 8:46 AM PDT 5/7/15.

Chris

And just to clarify for all … issue is with Visual Composer and *not* with the theme files; meaning 1.71B is not the culprit, nor was 1.7 — it’s entirely something with VC which causes major system issues when say a page is getting 15,000 hits at once; multiply that by 20-100 pages in a one hour period, and system throws a wrench.

1.4 was ROCK SOLID; 1.5 didn’t have major hiccups. Version of VC included with 1.6 started the issues for us. And version with 1.7 killed the box (or tried to anyway).

Nice work Radu/Chris,

Been lurking awaiting the fix but to echo the same experience, I’ve had to push our auto-scaling out to three hosts at visit levels that would normally be served by one. Back at 1.5 I was handling 2000 concurrent visitors on a single host 🙂

Update:
testing again today … managed to move another site from NM 1.6 and VC from Jan., to NM 1.71 with Shim Beta 1.02.

One minor issue with a full width (2 col) divider class on home page and full width text block (which I used to put some H1 text bottom of home page about the site).

Radu, sent you couple emails around 12pm PDT 5/11/15.

Presumably only downside to using the “shim” — is that one would need to switch VC back on to edit a page (e.g., static home page), and turn off the shim. I will be testing that usability here momentarily.

So, likely process would be
a) develop home page w/ VC turned on; shim OFF
b) switch off VC plugin
c) switch on TD shim plugin
d) check home page works properly

If one needed to edit
a) switch OFF the td shim beta
b) switch on VC
c) edit page, then repeat process in first block above.

Anecdotal for anybody wondering what is up with this beta testing 🙂

Not sure how mod_pagespeed would work with the swap out of VC vs shim …

==============
mod_pagespeed note:
for the first site I was testing shim with from last week;
I today switched back on mod_pagespeed (for those unaware, this is an Apache optimization plugin from Google which runs on server) and put back in all my htaccess optimizations and it’s working A-Okay (I had turned off all such optimizations for testing the SHIM plugin first try to avoid any conflicts with that).

• This reply was modified 11 years by simchris.
• This reply was modified 11 years by simchris.

Also, as an update — the above process with shim and using the VC version included with NM 1.6, and switching back and forth both work perfectly. Preumably this will work same way with current version of VC is you’re using it.

With the SHIM beta 1.02 plugin from TagDiv, only couple of blocks “so far” don’t work for home page

a) full width divider line for new row
b) raw html box
c) column text box

So, when using “shim” beta 1.02 these would reveal on home page

[vc_separator color=”grey” align=”align_center”]

[/vc_column_text]

and

[vc_raw_html]

Possibly others; I don’t use the whole kitchen sink; mostly normal content blocks.

I was using the column text element to put a full width h1 tag about website block for SEO.
For the time being to use shim plugin, I just moved that code to my custom sidebar on home page as normal text widget; since I’d already created custom CSS for the text anyway.

Hi,

is TD-SHIM meant for Newsmag only? I installed it on Newspaper and some shortcodes didn’t get through like Chris S noted, plus the following ones:
[rev_slider_vc alias=”slider4″ el_class=”zindex”]
[vc_empty_space height=”32px”]
[vc_raw_js]/*content*/[/vc_raw_js]

Everything else was working fine (for me), this is interesting and looks promising, thanks!

My presumption on this is that the shim will be for those folks like us who — for whatever reason — end up with a CPU spike of doom with current versions of Visual Composer running full time if we get upsurge in traffic — like 100x normal. For those without the CPU issue, likely not a big deal. So, this is a solution for that — but *not* a full-on replacement for VC.

From the hints that Radu has made, I think he has it in mind (well the TagDiv team) to come up with possible sideways solution — basically their own version of VC, essentially to not use the VC product at all, but with their own coding since I can imagine there would be some odd licensing issues and they are not the dishonest sort who would reverse engineer VC then bypass WP Bakery folk and use their tech. So, presumably they would come up with their own framework but MUCH “lighter” than how VC does it, where probably 70% of VC stuff isn’t used by most folk (in our case anyway).

Just my impression of where that might go.

I think this “fix” is there for those of us who need it but not a “hey everybody — stop using VC” — which is still needed to build out the layout for home page in first place.

Right now this just takes on the complaint I’ve had the past 18 months with VC, where I’ve often suggested a “design mode” and “deploy mode” … meaning “everything on” for design, then at deploy only use the 20% of stuff you’re actually using. Why load all that crap you don’t need with every load of home page. Kind of like having every option in Bootstrap but only using a quarter of the functions would be dumb. Same idea.

But I ramble ….
working for me so far, and CPU/server happy as a cat in a new blanket with paws full of catnip. Well, I might be exaggerating … 😉

Unfortunately I have to keep an eye on the CPU, I hardly have traffic upsurges but I’m stuck with a cheap host for now :/ Newspaper theme is stellar but needs resources ofc and I already have other robust plugins like qtranslate-x, events calendar or woocommerce, so everything helps until I move to a better place or maybe host the server myself. For now I’ll just try to make the most out of available optimizations, and most are yours Chris! many thanks btw! I had no time to implement everything yet but I had some good readings and learned more thanks to you.

I like VC and the pages it lets me build but a “deploy mode” sounds like something that should be a given. I don’t have many pages nor build new ones that often so VC main job for me is to render pages. VC doesn’t have to handle the biggest portion of the site’s content (articles) but handles the most important perhaps, the homepage, so efficiency in rendering should be important I think.