{"id":122,"date":"2007-05-19T00:25:13","date_gmt":"2007-05-18T16:25:13","guid":{"rendered":"http:\/\/wp.jiinjoo.com\/?p=122"},"modified":"2007-05-19T00:25:13","modified_gmt":"2007-05-18T16:25:13","slug":"what-is-happening-with-recom","status":"publish","type":"post","link":"https:\/\/wp.jiinjoo.com\/?p=122","title":{"rendered":"What is happening with ReCom"},"content":{"rendered":"

Urgh, I can’t even find a good way of explaining this on ReCom as the thing just won’t load for me, so I guess I’ll have to do this here for now. If you’re a geek and you want to help me resolve this problem read on.<\/p>\n

Currently ReCom<\/a> looks like this:<\/p>\n

<\/p>\n

There seems to be a problem with the MySQL server, sorry for the inconvenience.<\/p>\n

We should be back shortly.<\/strong><\/div>\n

Now what is actually happening behind the scenes, can be very quickly explained by my correspondence with the tech support:<\/p>\n\n\n
\n\n\n
\n\n\n
Intermittent connectivity to MySQL database<\/font><\/strong><\/span><\/td>\n<\/tr>\n<\/table>\n<\/td>\n<\/tr>\n<\/table>\n<\/td>\n<\/td>\n<\/tr>\n<\/table>\n
\nTicket Details<\/legend>\n\n\n\n\n
Ticket ID: <\/span><\/td>\nLYS-909847<\/span><\/td>\nDepartment: <\/span><\/td>\nSupport Divison<\/span><\/td>\n<\/tr>\n
Status: <\/span><\/td>\nOpen<\/font><\/span><\/td>\nPriority: <\/span><\/td>\nHigh<\/font><\/span><\/td>\n<\/tr>\n
Created On: <\/span><\/td>\n06 May 2007 10:14 AM<\/span><\/td>\nLast Update: <\/span><\/td>\n18 May 2007 11:43 AM<\/span><\/td>\n<\/tr>\n<\/table>\n<\/fieldset>\n
\nSH Account Information<\/legend>\n\n\n\n
Domain Name:<\/span><\/td>\nwww.recom.org<\/span><\/td>\n<\/tr>\n
Account Type<\/span><\/td>\nShared Hosting<\/span><\/td>\n<\/tr>\n<\/table>\n<\/fieldset>\n

<\/p>\n

Ong Jiin Joo<\/span> \t\tPosted On: 06 May 2007 10:14 AM<\/span><\/p>\n

Dear Sir\/Madam,<\/p>\n

Recently, our website www.recom.org<\/a> is experiencing intermittent connectivity to the database. This has been carrying on for 3 full days and we have not seen it resolve automatically.<\/p>\n

Please investigate and get back to us urgently. This has been affecting our loyal customers seriously and we would like to solve the issue. Nothing has changed in our software so we don’t believe it is a software error. It is more possible that the database connection pool has been used up of something.<\/p>\n

Please help!<\/p>\n

Thanks,
\n-Jiin Joo<\/p>\n

<\/span> \t \t \t \t\tNate H.<\/span> \t\t \t \t \t\t \t \t \t \t\t \t\tPosted On: 06 May 2007 10:30 AM<\/span><\/p>\n

Hello,<\/p>\n

MySQL was up and running, however, not properly. I have restarted this service which has corrected this issue. Your site is now resolving properly without errors. Please let us know if we can further assist you in this matter.<\/p>\n

Regards,
\nNate<\/span> \t \t \t \t\t \t \t \t\tOng Jiin Joo<\/span> \t\t \t \t \t\t \t \t \t \t\t \t\tPosted On: 07 May 2007 10:01 AM<\/span><\/p>\n

Hi Nate,<\/p>\n

The problem went away after you restarted it, but now it’s back again. Can you please verify that it is the same problem and identify the root cause?<\/p>\n

Very much appreciate it.<\/p>\n

Thanks,
\n-Jiin Joo<\/span> \t \t \t \t\t \t \t \t\tNate H.<\/span> \t\t \t \t \t\t \t \t \t \t\t \t\tPosted On: 07 May 2007 10:36 AM<\/span><\/p>\n

Hello,<\/p>\n

The root cause of this was a high amount of MySQL connections. I have restarted the service again, which will bump those users on the databases of the server that are no longer active, but that are still connected due to persistant connections enabled in someones script. Unfortunately, there is not much we can do about this, however, it should not be a recurring issue. I apologize that this has happened twice within the past few days.<\/p>\n

Regards,
\nNate<\/span> \t \t \t \t\tOng Jiin Joo<\/span> \t\t \t \t \t\t \t \t \t \t\t \t\tPosted On: 08 May 2007 05:34 AM<\/span><\/p>\n

Nate,<\/p>\n

No luck. The behavior continues on our end.<\/p>\n

We would like to help you identify if it is our own scripts that is hogging up all the connection. Can you dive in and find out this info? As I’m not a developer for this application, I need to gather all the information I can in order to advise the team. For example, it would help if you can identify the offending application and contact the owner of that application (if it happens to be recom.org, you should look for me), and either throttle that application on connection pool level or increase the connection pool size if there are no hardware limitations.<\/p>\n

This has been 5 days or more – shouldn’t it ring a bell in your data center already? I mean, there must be other applications running on this shared infrastructure right?<\/p>\n

HELP~~~!!!<\/p>\n

Thank you for your understanding.
\n-Jiin Joo<\/span> \t \t \t \t\t \t \t \t\t
\nMike Kr.<\/span> \t\t \t \t \t\t \t \t \t \t\t \t\tPosted On: 08 May 2007 06:25 AM<\/span><\/p>\n

Hello,<\/p>\n

The problem is your script is hitting the maximum number of simultaneous user connections to MySQL (the global limit is not being reached which is why only your site is affected). The current user limit is 25. Here is a list:<\/p>\n

root@sh78 [\/home\/recom24\/public_html]# mysqladmin processlist | grep recom24
\n| 1158 | recom24_recom | localhost | recom24_main2 | Sleep | 69914 | | |
\n| 28836 | recom24_recom | localhost | recom24_main2 | Sleep | 24603 | | |
\n| 30340 | recom24_recom | localhost | recom24_main2 | Sleep | 23361 | | |
\n| 30681 | recom24_recom | localhost | recom24_main2 | Sleep | 23108 | | |
\n| 30724 | recom24_recom | localhost | recom24_main2 | Sleep | 23062 | | |
\n| 31423 | recom24_recom | localhost | recom24_main2 | Sleep | 22566 | | |
\n| 31465 | recom24_recom | localhost | recom24_main2 | Sleep | 22542 | | |
\n| 31936 | recom24_recom | localhost | recom24_main2 | Sleep | 22155 | | |
\n| 32382 | recom24_recom | localhost | recom24_main2 | Sleep | 21721 | | |
\n| 33182 | recom24_recom | localhost | recom24_main2 | Sleep | 20980 | | |
\n| 34834 | recom24_recom | localhost | recom24_main2 | Sleep | 19218 | | |
\n| 35271 | recom24_recom | localhost | recom24_main2 | Sleep | 18740 | | |
\n| 36316 | recom24_recom | localhost | recom24_main2 | Sleep | 17728 | | |
\n| 36963 | recom24_recom | localhost | recom24_main2 | Sleep | 17210 | | |
\n| 38499 | recom24_recom | localhost | recom24_main2 | Sleep | 15814 | | |
\n| 38727 | recom24_recom | localhost | recom24_main2 | Sleep | 15617 | | |
\n| 39002 | recom24_recom | localhost | recom24_main2 | Sleep | 15395 | | |
\n| 39129 | recom24_recom | localhost | recom24_main2 | Sleep | 15281 | | |
\n| 39262 | recom24_recom | localhost | recom24_main2 | Sleep | 15151 | | |
\n| 39265 | recom24_recom | localhost | recom24_main2 | Sleep | 15160 | | |
\n| 39266 | recom24_recom | localhost | recom24_main2 | Sleep | 15145 | | |
\n| 44598 | recom24_recom | localhost | recom24_main2 | Sleep | 9121 | | |
\n| 44630 | recom24_recom | localhost | recom24_main2 | Sleep | 9087 | | |
\n| 53130 | recom24_recom | localhost | recom24_main2 | Sleep | 747 | | |
\n| 53137 | recom24_recom | localhost | recom24_main2 | Sleep | 724 | |<\/p>\n

This is usually caused by improper usage of persistent connections in PHP scripts. I have restarted the server again to clear out all of the stale connections, but this is an issue you should bring up with your developers.<\/p>\n

Regards,
\nMike<\/span>Okay, anyone got it? If not read on.<\/p>\n

ReCom runs phpNuke<\/a>, a highly modified version due to the immense amount of feature set that was required by earlier active members. Today the bulk of the traffic goes to the Forum and other associated communication tools. Recently the number of visitors (members \/ non-members alike) hitting the site has been on the increase to the extent that we’re now seeing very observable peak load during scholarship season.<\/p>\n

Unfortunately, due to nature of shared hosting, each application is only allowed up to 25 connections to the database. Now these are concurrent connections, which means that when you visit the site, you don’t actually “hold” one of the connection hostage, but you basically acquire one of these when you click on something until the page is loaded. In the normal circumstances, the connection is then released back into the connection pool so that other applications on the same shared host can grab it.<\/p>\n

However, there’s a catch, which is the way PHP implements Persistent Connections. I’ll leave the academic exercise of understanding what persistent connections<\/a> means to you. Our initial speculation is that our version of phpNuke actually does that, i.e. when one is done loading a particular page, the connection is not released by the application, but handed to the other thread that’s handling another page for another user. This is usually done to increase performance of MySQL, as any Database Administrators (DBA) or Computer Science (CS) students would be able to tell you that locking yourself up to wait for a DB connection from the connection pool is usually a very expensive operation.<\/p>\n

There is a downside to doing this, i.e. when you’re actually _not_ using it, the PHP engine might hold on to the connection for a specific timeout period before releasing the connection. So there is a chance that MySQL’s performance is actually worse as non of the connections are available.<\/p>\n

However<\/strong>, as Luke has painstakingly went through the code, ReCom does not<\/strong> use persistent connections. This means that the snapshot that tech support gave us can only mean one thing – that at anyone time there are actually more than 25 concurrent users loading a page. This on a system architecture perspective is ridiculous. I run systems for thousand man companies and they load more than just a simple forum page and yet they only need 5 concurrent connections. Or quoting some example for MySQL website<\/a>:<\/p>\n

As with all other configuration rules-of-thumb, the answer is “It depends.” While the optimal size depends on anticipated load and average database transaction time, the optimum connection pool size is smaller than you might expect. If you take Sun’s Java Petstore blueprint application for example, a connection pool of 15-20 connections<\/strong> can serve a relatively moderate load (600 concurrent users<\/strong>) using MySQL and Tomcat with response times that are acceptable.<\/p><\/blockquote>\n

(FYI, the petstore application is just a generic application – think about buying pets online in a typical e-commerce site – it should have a higher browsing rate than ReCom, where people tend to stop and read instead of look at pet pictures)<\/p>\n

I don’t want to be the one to “declare” that ReCom is really getting the popularity it cannot handle, because I didn’t do the diagnostic myself (yet?), but I do think the ReCom Anchors owe many of our loyal members an explanation. This is as detail as I can give now, and with this I’m begging for volunteer resource to help me with this issue. Basically we need to track down exactly how phpNuke close connection just in case there’re alternate code paths in the system that causes connections to be left open (*gasp*) or ascertain (i.e. collect statistics) the right sizing for the servers required to handle the load. There is also a remote possibility that our web hosting company simply sucks (e.g. implemented a buggy version of MySQL or PHP) but with no facts at hand we can’t decide on the best course of action yet.<\/p>\n

Thank you for reading.<\/p>\n

p\/s I know there are a lot of alternative energy – e.g. let’s switch to another CMS, let’s visit another forum etc. Let me assure you that since it is a technical issue, we would treat it as a technical issue, or should I say, we should<\/em> treat it as a technical issue. Moreover, ReCom’s platform is a multi-year investment, and is not easily replicable by just “downloading” another CMS and pressing the play button. I’ll organize system architecture classes if you want to understand more. \ud83d\ude1b<\/p>\n","protected":false},"excerpt":{"rendered":"

Urgh, I can’t even find a good way of explaining this on ReCom as the thing just won’t…<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-122","post","type-post","status-publish","format-standard","hentry","category-diary","content-wrap"],"_links":{"self":[{"href":"https:\/\/wp.jiinjoo.com\/index.php?rest_route=\/wp\/v2\/posts\/122","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wp.jiinjoo.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wp.jiinjoo.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wp.jiinjoo.com\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/wp.jiinjoo.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=122"}],"version-history":[{"count":0,"href":"https:\/\/wp.jiinjoo.com\/index.php?rest_route=\/wp\/v2\/posts\/122\/revisions"}],"wp:attachment":[{"href":"https:\/\/wp.jiinjoo.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=122"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wp.jiinjoo.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=122"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wp.jiinjoo.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=122"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}